Skip to content

Make a Cosmo Tech Simulator cloud-ready

In this tutorial we will take our updated Brewery Simulator project as we left it in the previous tutorial and add modification to get it ready to be used inside a Cosmo Tech Platform.

Requirements

  • docker
  • A Cosmo Tech Platform minimal version of 3.0
  • A Cosmo Tech SDK minimal version of 10.0
  • Installed SDK package must include csmcli

How does the Cosmo Tech Platform runs orchestration?

The Cosmo Tech Platform will run pre-registered docker images based on the definition of Solutions in its API.

In the API you can define a Solution that will refer to a specific version of a docker image in a given repository. That Solution defines sets of Parameters and Run Templates. Then Workspaces are defined in the API and point to a given Solution allowing you to create a Scenario which will tie together Datasets, Parameters, and a Run Template which will then be run by the API.

Datasets are a description of how and where data are stored, Parameters can be simple types or full Datasets allowing you to update the main Dataset of your Scenario, therefore changing its initial state or behavior (or both).

Running your Scenario will create a ScenarioRun into the API and that ScenarioRun will make use of the docker image your previously defined. The API will call the image with an Entrypoint called entrypoint.py and that call should be your full orchestration.

Why entrypoint.py and not csm-orc ?

The definition of the Entrypoint being entrypoint.py is left for compatibility with old images on the Cosmo Tech API (pre 3.0). It will be changed in the future, but for now csm-orc comes bundled with a script called entrypoint.py that allows both legacy and new runs of the docker image.

How does entrypoint.py works?

The entrypoint.py command (and its equivalent csm-orc entrypoint) works using some Environment variables to switch between 3 modes:

  • The general entrypoint with no special configuration that allow to run your run templates requiring the environment variable CSM_RUN_TEMPLATE_ID to be set with your run template name.
  • The legacy entrypoint activated by setting the environment variable CSM_ENTRYPOINT_LEGACY to true which is made available for compatibility with pre 3.0 Cosmo Tech API.
  • The direct simulator mode which requires the environment variable CSM_RUN_TEMPLATE_ID to not be set, and will then run the simulator by passing it any argument of the command.

Every Environment Variable passed to the command will be forwarded to the csm-orc run command inside, and the command will be run with the working directory set to /pkg/share.

Which Environment Variables are made available by the API?

The Cosmo Tech API will forward a set of environment variables to any Simulator containers. You can find the full list in the following table.

List of environment variables
Environment Variable Description
AZURE_CLIENT_ID An identifier to an Azure identity defined during the installation of the platform
AZURE_CLIENT_SECRET A secret tied to the given client ID allowing to connect with it
AZURE_TENANT_ID An identifier of your Azure tenant to be able to connect to it
IDENTITY_PROVIDER Will be set to azure in an Azure based API
CSM_API_URL The url to the Cosmo Tech API your are connecting to
CSM_API_SCOPE An identifier scope used to get permission to your API
CSM_DATASET_ABSOLUTE_PATH A local folder path to which an external volume is mounted to and in which you can write datasets
CSM_PARAMETERS_ABSOLUTE_PATH A local folder path to which an external volume is mounted to and in which you can write parameters
TWIN_CACHE_HOST The URL to the twin cache service inside the platform (deprecated)
TWIN_CACHE_PORT The port to the twin cache service inside the platform (deprecated)
TWIN_CACHE_PASSWORD The password to the twin cache service inside the platform (deprecated)
TWIN_CACHE_USERNAME The username to the twin cache service inside the platform (deprecated)
CSM_SIMULATION_ID The Simulation ID for your current scenario run
AZURE_DATA_EXPLORER_RESOURCE_URI The URI used to query an Azure Data Explorer Cluster tied to your Workspace
AZURE_DATA_EXPLORER_RESOURCE_INGEST_URI The URI used to send data to an Azure Data Explorer Cluster tied to your Workspace
AZURE_DATA_EXPLORER_DATABASE_NAME The name of the database of your Workspace in Azure Data Explorer
CSM_ORGANIZATION_ID The ID of your Organization in the Cosmo Tech API
CSM_WORKSPACE_ID The ID of your Workspace in the Cosmo Tech API
CSM_SCENARIO_ID The ID of your Scenario in the Cosmo Tech API
CSM_RUN_TEMPLATE_ID The ID of your Run Template in the Cosmo Tech API (as defined in your Solution)
CSM_CONTAINER_MODE Will always be csm-orc left for compatibility with pre 3.0 API
CSM_ENTRYPOINT_LEGACY Will always be false
CSM_PROBES_MEASURES_TOPIC An amqp address used by the simulator to send your probe data to ADX via Event Hub
CSM_CONTROL_PLANE_TOPIC An amqp address used by the simulator to send your control info to ADX via Event Hub (mostly number of measures sent)
CSM_SIMULATION The Simulation file that should be used to run your simulation (this value is defined in your Run Template and targets one of the files in the Simulation/ folder)

Connect to the API to get our Scenario data

Multiple ways exists to connect to the API and query some data, but for simplicity there exists a command in csm-data that will make use of known API environment variables and do the work for you.

What is csm-data

csm-data is a data oriented CLI part of the library CosmoTech-Acceleration-Library (abreviated as CoAL)

It is made to have pre-made commands to facilitate use of most of the services a modelisator/integrator could require while working on a run template

You can get it by installing CoAL starting with version 0.7.0

How to install CoAL and csm-data
pip install CosmoTech-Acceleration-Library~=0.7.0

That command is csm-data api scenariorun-load-data (documentation of the command is available here).

The command makes use of 5 environment variables set by the API (as described in the previous section):

  • CSM_ORGANIZATION_ID
  • CSM_WORKSPACE_ID
  • CSM_SCENARIO_ID
  • CSM_DATASET_ABSOLUTE_PATH
  • CSM_PARAMETERS_ABSOLUTE_PATH

And uses 3 control environment variables:

  • WRITE_JSON: If set to true will write a parameters.json file in CSM_PARAMETERS_ABSOLUTE_PATH.
  • WRITE_CSV: If set to true will write a parameters.csv file in CSM_PARAMETERS_ABSOLUTE_PATH.
  • FETCH_DATASET: If set to true will download all the Datasets tied to your scenario, will write the main ones (not defined as a Parameter) in CSM_DATASET_ABSOLUTE_PATH and will write the others in CSM_PARAMETERS_ABSOLUTE_PATH/[parameter name] where [parameter name] is the name of the Parameter targeting the Dataset (with a type set to %DATASETID%).

With that command one can easily download any dataset defined in the Cosmo Tech API, as long as those datasets use one of the standard connections defined in the API (as of version 3.0: Azure Blob Storage, Azure Digital Twin, TwinGraph Storage).

Combine everything in a Run Template

Using all those new information we can see that most of the actions needed to run our code based on API data are already prepackaged.

We can:

  • Download our scenario information using csm-data api scenariorun-load-data.
  • Apply our parameters with our apply_parameters.py.
  • Run our simulation using csm-orc run-step.
  • And send our simulation results to an external system (here Azure Data Explorer) by setting environment variables during the run-step.

It is then easy to update our previous run.json to take those changes into account.

New code/run_templates/orchestrator_tutorial_2 folder

To keep a clean distinction on the code of the previous tutorial and this one, we will refer to the folder code/run_templates/orchestrator_tutorial_2 which is an exact copy of the folder code/run_templates/orchestrator_tutorial_1.

You can create it by using the following command:

copy `orchestrator_tutorial_1`
cp -r code/run_templates/orchestrator_tutorial_1 code/run_templates/orchestrator_tutorial_2

code/run_templates/orchestrator_tutorial2/run.json
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
{
  "steps": [
    {
      "id": "DownloadScenarioData",
      "command": "csm-data",
      "description": "Connect to the Cosmo Tech API to download dataset and parameters",
      "arguments": [
        "api", 
        "scenariorun-load-data",
        "--write-json"
      ],
      "useSystemEnvironment": true,
      "environment": {
        "CSM_API_URL": {
          "description": "The URL used to query your Cosmo Tech API instance"
        },
        "CSM_API_SCOPE": {
          "description": "The scope of identification used to request access token for your Cosmo Tech API instance"
        },
        "CSM_ORGANIZATION_ID": {
          "description": "The id of the organization in the Cosmo Tech API"
        },
        "CSM_WORKSPACE_ID": {
          "description": "The id of the workspace in the Cosmo Tech API"
        },
        "CSM_SCENARIO_ID": {
          "description": "The id of the scenario in the Cosmo Tech API"
        },
        "CSM_DATASET_ABSOLUTE_PATH": {
          "description": "The path used to store the datasets of the scenario"
        },
        "CSM_PARAMETERS_ABSOLUTE_PATH": {
          "description": "The path used to store the parameters of the scenario"
        }
      }
    },
    {
      "id": "ApplyParameters",
      "command": "python",
      "arguments": [
        "code/run_templates/orchestrator_tutorial_2/apply_parameters.py",
        "$CSM_DATASET_ABSOLUTE_PATH",
        "$CSM_DATASET_ABSOLUTE_PATH",
        "$CSM_PARAMETERS_ABSOLUTE_PATH/parameters.json"
      ],
      "description": "Apply our parameters to the dataset",
      "environment": {
        "CSM_DATASET_ABSOLUTE_PATH": {
          "description": "The path to the dataset to update",
          "defaultValue": "Simulation/Resource/scenariorun-data"
        },
        "CSM_PARAMETERS_ABSOLUTE_PATH": {
          "description": "The path to a folder containing our parameters.json file",
          "defaultValue": "code/run_templates/orchestrator_tutorial_2"
        }
      },
      "precedents": [
        "DownloadScenarioData"
      ]
    },
    {
      "id": "SimulationRun",
      "command": "csm-orc",
      "arguments": [
        "run-step",
        "--template",
        "orchestrator_tutorial_2",
        "--steps",
        "engine"
      ],
      "description": "Runs the simulation targeted by CSM_SIMULATION",
      "useSystemEnvironment": true,
      "environment": {
        "CSM_SIMULATION": {
          "description": "The simulation file to run",
          "defaultValue": "BusinessApp_Simulation"
        }
      },
      "precedents": [
        "ApplyParameters"
      ]
    }
  ]
}

We can see a few changes and additions compared to the previous run.json file:

  • We created a new step called DownloadScenarioData that makes use of csm-data api scenariorun-load-data:
    • In this step we defined a few environment variables required to run it, but we still added useSystemEnvironment to true to ensure any environment variable required to connect to the API is made available as well.
  • In the ApplyParameters we made a few changes:
    • DATASET/PARAMETERS_PATH became CSM_DATASET/PARAMETERS_ABSOLUTE_PATH.
    • We added /parameters.json in the arguments for the parameters path.
    • We added a precendent step to schedule it after the DownloadScenarioData.
  • In the SimulatorRun step changes are minimal:
    • The template targeted is the new orchestrator_tutorial_2.
    • The default value of CSM_SIMULATION has been changed to BusinessApp_Simulation since docker simulation can't make use of visual consumers.

Build a docker image

Now that we have our run.json ready we can build our docker image.

Build a docker image using csmcli
# First we will clean everything to ensure a correct generation of the project
csm clean
# Now we can build our full project
csm flow
# The simulator is now ready to be packaged
csm docker build

After running those commands we end up with a docker image cosmotech/mybrewery_simulator ready to be used.

A simple test can be made with the following command:

Test the docker image
docker run -e CSM_RUN_TEMPLATE_ID=orchestrator_tutorial_1 -e CSM_SIMULATION=BusinessApp_Simulation -v ./Simulation/Resource/scenariorun-data:/mnt/scenariorun-data cosmotech/mybrewery_simulator

About the parameters of the command docker run

  • -e sets an environment variable during the docker run, here we set our run template id and our simulation.
  • -v sets a volume during the docker run, here we mount our local Simulation/Resource/scenariorun-data folder to the path /mnt/scenariorun-data in the docker container, allowing us to use our existing dataset during the run.

Define API items

The following sections give you simple examples of .yaml files used to define resources in the Cosmo Tech API.

Define a Solution

An example of Solution.yaml (a file used by the Cosmo Tech API) for our current solution would be:

Solution.yaml
key: "brewery"
name: "Demo Brewery Solution, csm-orc tutorial"
description: "A simple example of a Cosmo Tech solution built using the Brewery model"
tags:
  - "csm-orc"
  - "Brewery"
repository: "mybrewery_simulator"
version: "0.0.1"
parameters:
  - id: "Stock"
    varType: "int"
    defaultValue: 100
  - id: "RestockQty"
    varType: "int"
    defaultValue: 25
  - id: "NbWaiters"
    varType: "int"
    defaultValue: 5
parameterGroups:
  - id: bar_parameters
    parameters:
      - Stock
      - RestockQty
      - NbWaiters
runTemplates:
  - id: "orchestrator_tutorial_2"
    name: "Basic what-if"
    description: "Basic what-if simulation"
    parameterGroups:
      - bar_parameters
    csmSimulation: BusinessApp_Simulation

Full Open API description of a Solution is available here.

Open API description of a Solution (components.schemas) 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
    Solution:
      type: object
      x-class-extra-annotation: "@com.redis.om.spring.annotations.Document"
      description: a version of a Solution
      properties:
        id:
          type: string
          x-field-extra-annotation: "@org.springframework.data.annotation.Id"
          readOnly: true
          description: the Solution version unique identifier
        organizationId:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: string
          readOnly: true
          description: the Organization unique identifier
        key:
          type: string
          description: the Solution key which group Solution versions
        name:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          description: the Solution name
        description:
          type: string
          description: the Solution description
        repository:
          type: string
          description: the registry repository containing the image
        csmSimulator:
          type: string
          description: the main Cosmo Tech simulator name used in standard Run Template
        version:
          type: string
          description: the Solution version MAJOR.MINOR.PATCH. Must be aligned with an existing repository tag
        ownerId:
          type: string
          readOnly: true
          description: the User id which own this Solution
        sdkVersion:
          type: string
          description: the MAJOR.MINOR version used to build this solution
        url:
          type: string
          description: an optional URL link to solution page
        tags:
          type: array
          description: the list of tags
          items:
            type: string
        parameters:
          type: array
          description: the list of Run Template Parameters
          items:
            $ref: '#/components/schemas/RunTemplateParameter'
        parameterGroups:
          type: array
          description: the list of parameters groups for the Run Templates
          items:
            $ref: '#/components/schemas/RunTemplateParameterGroup'
        runTemplates:
          type: array
          description: list of Run Template
          items:
            $ref: '#/components/schemas/RunTemplate'
        security:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          allOf:
            - $ref: '#/components/schemas/SolutionSecurity'
  #       required:
  #         - key
  #         - name
  #         - repository
  #         - version
  #         - runTemplates
    SolutionSecurity:
      type: object
      description: the Solution security information
      properties:
        default:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the role by default
        accessControlList:
          type: array
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the list which can access this Solution with detailed access control information
          items:
            $ref: '#/components/schemas/SolutionAccessControl'
      required:
        - default
        - accessControlList
    SolutionAccessControl:
      type: object
      description: a Solution access control item
      properties:
        id:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the identity id
        role:
          type: string
          description: a role
      required:
        - id
        - role
    SolutionRole:
      type: object
      description: the Solution Role
      properties:
        role:
          type: string
          description: the Solution Role
      required:
        - role
    RunTemplate:
      type: object
      description: a Solution Run Template
      properties:
        id:
          type: string
          description: the Solution Run Template id
        name:
          type: string
          description: the Run Template name
          deprecated: true
        labels:
          $ref: '#/components/schemas/TranslatedLabels'
        description:
          type: string
          description: the Run Template description
        csmSimulation:
          type: string
          description: the Cosmo Tech simulation name. This information is send to the Engine. Mandatory information if no Engine is defined
        tags:
          type: array
          description: the list of Run Template tags
          items:
            type: string
        computeSize:
          type: string
          description: the compute size needed for this Run Template. Standard sizes are basic and highcpu. Default is basic
        runSizing:
          description: definition of resources needed for the scenario run
          $ref: "#/components/schemas/RunTemplateResourceSizing"
        noDataIngestionState:
          type: boolean
          description: set to true if the run template does not want to check data ingestion state (no probes or not control plane)
        fetchDatasets:
          type: boolean
          description: whether or not the fetch dataset step is done
        scenarioDataDownloadTransform:
          type: boolean
          description: whether or not the scenario data download transform step step is done
        fetchScenarioParameters:
          type: boolean
          description: whether or not the fetch parameters step is done
        applyParameters:
          type: boolean
          description: whether or not the apply parameter step is done
        validateData:
          type: boolean
          description: whether or not the validate step is done
        sendDatasetsToDataWarehouse:
          type: boolean
          description: whether or not the Datasets values are send to the DataWarehouse prior to Simulation Run. If not set follow the Workspace setting
        sendInputParametersToDataWarehouse:
          type: boolean
          description: whether or not the input parameters values are send to the DataWarehouse prior to Simulation Run. If not set follow the Workspace setting
        preRun:
          type: boolean
          description: whether or not the pre-run step is done
        run:
          type: boolean
          description: whether or not the run step is done
        postRun:
          type: boolean
          description: whether or not the post-run step is done
        parametersJson:
          type: boolean
          description: whether or not to store the scenario parameters in json instead of csv
        parametersHandlerSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        datasetValidatorSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        preRunSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        runSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        postRunSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        scenariodataTransformSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        parameterGroups:
          type: array
          description: the ordered list of parameters groups for the Run Template
          items:
            type: string
            description: a Run Template Group Parameter id
        stackSteps:
          type: boolean
          description: whether or not to stack adjacent scenario run steps in one container run which will chain steps
        gitRepositoryUrl:
          type: string
          description: an optional URL to the git repository
        gitBranchName:
          type: string
          description: an optional git branch name
        runTemplateSourceDir:
          type: string
          description: an optional directory where to find the run template source
        orchestratorType:
          $ref: "#/components/schemas/RunTemplateOrchestrator"
        executionTimeout:
          type: integer
          description: an optional duration in seconds in which a workflow is allowed to run
        deleteHistoricalData:
          $ref: "#/components/schemas/DeleteHistoricalData"
      required:
        - id
#         - name
    DeleteHistoricalData:
      type: object
      description: Configuration of scenario runs deletion automatic mecanism
      properties:
        enable:
          type: boolean
          description: Activate or Deactivate historical data deletion
          default: true
        pollFrequency:
          type: integer
          description: define the polling frequency of scenario run status (in millis)
          default: 10000
        timeOut:
          type: integer
          description: define the polling timeout
          default: 28800
      required:
        - enable

    RunTemplateOrchestrator:
      type: string
      description: the Orchestrator to use for the Run Template
      enum:
        - argoWorkflow
        - csmOrc
    RunTemplateHandlerId:
      type: string
      description: the Run Template step handler identifier
      enum: ["parameters_handler","validator","prerun","engine","postrun","scenariodata_transform"]
    RunTemplateStepSource:
      type: string
      description: the source local, cloud or git repository
      enum: ["cloud","git","local","platform"]
    RunTemplateResourceSizing:
      type: object
      description: a description object for resource requests and limits (default same configuration as basic sizing)
      properties:
        requests:
          $ref: '#/components/schemas/ResourceSizeInfo'
        limits:
          $ref: '#/components/schemas/ResourceSizeInfo'
      required:
        - requests
        - limits
    ResourceSizeInfo:
      type: object
      description: define cpus and memory needs
      properties:
        cpu:
          type: string
          description: define cpu needs
        memory:
          type: string
          description: define memory needs
      required:
        - cpu
        - memory
    RunTemplateParameterGroup:
      type: object
      description: a Parameter Group for a Run Template
      properties:
        id:
          type: string
          description: the Parameter Group id
        labels:
          $ref: '#/components/schemas/TranslatedLabels'
        isTable:
          type: boolean
          description: does the group define a table
        options:
          type: object
          description: freeform options
          additionalProperties: true
        parentId:
          type: string
          description: the Run Template Group parent Id
        parameters:
          type: array
          description: an ordered list of Run Template Parameters
          items:
            type: string
            description: a Run Template Parameter id
      required:
        - id
#         - labels
#         - parameters
    RunTemplateParameter:
      type: object
      description: a Run Template Parameter
      properties:
        id:
          type: string
          description: the Parameter id
        labels:
          $ref: '#/components/schemas/TranslatedLabels'
        varType:
          type: string
          description: the variable type for the parameter. Basic types or special type %DATASETID%
        defaultValue:
          type: string
          description: the default value for this parameter
        minValue:
          type: string
          description: the minimum value for this parameter
        maxValue:
          type: string
          description: the maximum value for this parameter
        regexValidation:
          type: string
          description: a regex to validate the value
        options:
          type: object
          description: freeform options
          additionalProperties: true
      required:
        - id
#         - labels
#         - varType
    TranslatedLabels:
      type: object
      description: a translated label with key as ISO 639-1 code
      additionalProperties:
        type: string

Once this file is sent to a Cosmo Tech API we will get a Solution ID of the form sol-xxxxxxxx which will allow us to create further elements inside the API referencing our solution.

Now let's look deeper at how to create our Solution.yaml.

Solution description

Solution.yaml - description
key: "brewery"
name: "Demo Brewery Solution, csm-orc tutorial"
description: "A simple example of a Cosmo Tech solution built using the Brewery model"
tags:
  - "csm-orc"
  - "Brewery"
repository: "mybrewery_simulator"
version: "0.0.1"

We can see a few important properties in this part of the Solution file:

  • key: it is a "grouping" value used to make multiple Solution using the same base but different versions together.
  • name: a simple name given to the Solution.
  • description: a simple description of the Solution.
  • tags: a list of tags used in the API to filter Solution resources.
  • repository: the name of the docker image inside the image registry.
  • version: the version of the docker image tied to the Solution.

parameters and parameterGroups

Solution.yaml - parameters
parameters:
  - id: "Stock"
    varType: "int"
    defaultValue: 100
  - id: "RestockQty"
    varType: "int"
    defaultValue: 25
  - id: "NbWaiters"
    varType: "int"
    defaultValue: 5
parameterGroups:
  - id: bar_parameters
    parameters:
      - Stock
      - RestockQty
      - NbWaiters

In this part we declare our scenario parameters. As previously decided we need three parameters:

  • NbWaiters
  • RestockQty
  • Stock

Each parameter can be given a type (property varType) and a default value (property 'defaultValue'). Then we grouped our parameters in a parametersGroup "bar_parameters". Parameter groups will make our parameters available in further part of the Solution.

runTemplates

Solution.yaml - runTemplates
runTemplates:
  - id: "orchestrator_tutorial_2"
    name: "Basic what-if"
    description: "Basic what-if simulation"
    parameterGroups:
      - bar_parameters
    csmSimulation: BusinessApp_Simulation

In this part we finally define our Run Template for the API. We have three main elements to define:

  • id: the id of a run template is the name of a folder inside code/run_templates that will contain a run.json file, along with one sub-folder for each step in the run template.
  • parameterGroups: list of parameter groups available for the run template. Parameters declared in those groups can be changed by users in order to create new scenarios.
  • csmSimulation: the name of a simulation file (without the .sml.xml suffix), to be set in the CSM_SIMULATION_ID environment variable. This simulation will run as part of the run template execution.

Define a Workspace

An example of Workspace.yaml which is a file used by the Cosmo Tech API for our current solution would be:

Workspace.yaml
key: brewery_orchestrator_tutorial
name: Brewery tutorial workspace
solution:
  solutionId: "sol-xxxxxxxx"

Full Open API description of a Workspace is available here.

Open API description of a Workspace (components.schemas) 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
    Workspace:
      type: object
      x-class-extra-annotation: "@com.redis.om.spring.annotations.Document"
      description: a Workspace
      properties:
        id:
          type: string
          x-field-extra-annotation: "@org.springframework.data.annotation.Id"
          readOnly: true
          description: Workspace unique identifier generated by the API
          example: "w-0123456789ab"
        organizationId:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: string
          readOnly: true
          description: Organization unique identifier under which the workspace resides
          example: "o-0123456789ab"
        key:
          type: string
          description: technical key for resource name convention and version grouping. Must be unique
          example: "MyKey"
        name:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          description: Workspace name. This name is display in the sample webApp
          example: "FranceOffice"
        description:
          type: string
          description: the Workspace description
        linkedDatasetIdList:
          type: array
          description: list of dataset linked to this dataset
          items:
            type: string
        version:
          type: string
          description: the Workspace version MAJOR.MINOR.PATCH.
          example: "1.0.0"
        tags:
          type: array
          description: the list of tags
          items:
            type: string
        ownerId:
          type: string
          readOnly: true
          description: the user id which own this workspace. set by the API from Authentification service receved id
        solution:
          $ref: '#/components/schemas/WorkspaceSolution'
        webApp:
          $ref: '#/components/schemas/WorkspaceWebApp'
        sendInputToDataWarehouse:
          type: boolean
          description: default setting for all Scenarios and Run Templates to set whether or not the Dataset values and the input parameters values are send to the DataWarehouse prior to the ScenarioRun
        useDedicatedEventHubNamespace:
          type: boolean
          default: false
          description: Set this property to true to use a dedicated Azure Event Hub Namespace for this Workspace.
            The Event Hub Namespace must be named \'<organization_id\>-<workspace_id\>\' (in lower case).
            This Namespace must also contain two Event Hubs named \'probesmeasures\' and \'scenariorun\'.
        dedicatedEventHubSasKeyName:
          type: string
          description: the Dedicated Event Hub SAS key name, default to RootManageSharedAccessKey. Use the /secret endpoint to set the key value
        dedicatedEventHubAuthenticationStrategy:
          type: string
          description: the Event Hub authentication strategy, SHARED_ACCESS_POLICY or TENANT_CLIENT_CREDENTIALS. Default to the one defined for the tenant.
        sendScenarioRunToEventHub:
          type: boolean
          default : true
          description: default setting for all Scenarios and Run Templates to set whether or not the ScenarioRun is send to the Event Hub
        sendScenarioMetadataToEventHub:
          type: boolean
          default: false
          description: Set this property to false to not send scenario metada to Azure Event Hub Namespace for this Workspace.
            The Event Hub Namespace must be named \'<organization_id\>-<workspace_id\>\' (in lower case).
            This Namespace must also contain two Event Hubs named \'scenariometadata\' and \'scenariorunmetadata\'.
        datasetCopy:
          type: boolean
          default: true
          description: Activate the copy of dataset on scenario creation, meaning that each scenario created in this workspace
            will make this copy. when false, scenario use directly the dataset specified.
        security:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          allOf:
           - $ref: '#/components/schemas/WorkspaceSecurity'
      required:
        - key
        - name
        - solution
    WorkspaceSecurity:
      type: object
      description: the workspace security information
      properties:
        default:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the role by default
        accessControlList:
          type: array
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the list which can access this Workspace with detailed access control information
          items:
            $ref: '#/components/schemas/WorkspaceAccessControl'
      required:
        - default
        - accessControlList
    WorkspaceAccessControl:
      type: object
      description: a Workspace access control item
      properties:
        id:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the identity id
        role:
          type: string
          description: a role
      required:
        - id
        - role
    WorkspaceRole:
      type: object
      description: the Workspace Role
      properties:
        role:
          type: string
          description: the Workspace Role
      required:
        - role
    WorkspaceFile:
      type: object
      description: a Workspace File resource
      properties:
        fileName:
          type: string
          description: the Workspace File name
    WorkspaceWebApp:
      type: object
      description: a Workspace Web Application
      properties:
        url:
          type: string
          description: the Workspace Web Application URL
        iframes:
          type: object
          description: a map of iframeKey/iframeURL
          additionalProperties: true
        options:
          type: object
          description: free form options for Web Application
          additionalProperties: true
      required:
        - url
    WorkspaceSolution:
      type: object
      description: the Workspace Solution configuration
      properties:
        solutionId:
          type: string
          description: the Solution Id attached to this workspace
        runTemplateFilter:
          type: array
          description: the list of Solution Run Template Id to filter
          items:
            type: string
        defaultRunTemplateDataset:
          type: object
          description: a map of RunTemplateId/DatasetId to set a default dataset for a Run Template
          additionalProperties: true
      # required:
        # - solutionId
    WorkspaceSecret:
      type: object
      description: the secret definition
      properties:
        dedicatedEventHubKey:
          type: string
          description: the dedicated event hub shared access key

After creating a Workspace resource using the API you will need to create additional resources for it in your platform if you want to run scenarios.

Once your Workspace is created you will get a workspace identifier of the form w-xxxxxxxx. It will allow you to connect to your workspace and reference it in other resources.

We have 3 required parts in a Workspace:

  • key: a unique identifier for your workspace which will be used during resource creation.
  • name: the name of your Workspace.
  • solution.solutionId: the solution identifier (sol-xxxxxxxx) for the solution you want to use in your workspace.

Define a Dataset

An example of Dataset.yaml which is a file used by the Cosmo Tech API for our current solution would be:

Dataset.yaml
name: Orchestrator Tutorial Dataset
description: Example dataset defined during the Orchestrator Tutorial
tags:
  - Brewery
  - Tutorial

Full Open API description of a Dataset is available here.

Open API description of a Dataset (components.schemas) 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
    FileUploadValidation:
      type: object
      description: files read on upload
      properties:
        nodes:
          type: array
          description: list of filename found on nodes folder
          items:
            $ref: '#/components/schemas/FileUploadMetadata'
        edges:
          type: array
          description: list of filename found on edges folder
          items:
            $ref: '#/components/schemas/FileUploadMetadata'
    FileUploadMetadata:
      type: object
      properties:
        name:
          type: string
        size:
          type: integer
    DatasetSearch:
      type: object
      description: the search options
      properties:
        datasetTags:
          type: array
          description: the dataset tag list to search
          items:
            type: string
      required:
        - datasetTags
    Dataset:
      type: object
      x-class-extra-annotation: "@com.redis.om.spring.annotations.Document"
      description: a Dataset
      properties:
        id:
          x-field-extra-annotation: "@org.springframework.data.annotation.Id"
          type: string
          readOnly: true
          description: the Dataset unique identifier
        name:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          type: string
          description: the Dataset name
        description:
          type: string
          description: the Dataset description
        ownerId:
          type: string
          readOnly: true
          description: the User id which own this Dataset
        ownerName:
          type: string
          readOnly: true
          description: the name of the owner
        organizationId:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: string
          readOnly: true
          description: the Organization Id related to this Dataset
        parentId:
          type: string
          description: the Dataset id which is the parent of this Dataset
        linkedWorkspaceIdList:
          type: array
          description: list of workspace linked to this dataset
          items:
            type: string
        twingraphId:
          type: string
          description: the twin graph id
        main:
          type: boolean
          description: is this the main dataset
        creationDate:
          type: integer
          format: int64
          readOnly: true
          description: the Dataset creation date
        refreshDate:
          type: integer
          format: int64
          readOnly: true
          description: the last time a refresh was done
        sourceType:
          $ref: '#/components/schemas/DatasetSourceType'
        source:
          $ref: '#/components/schemas/SourceInfo'
        ingestionStatus:
          type: string
          description: the Dataset ingestion status
          enum:
            - NONE
            - PENDING
            - ERROR
            - SUCCESS
        twincacheStatus:
          type: string
          description: the twincache data status
          enum:
            - EMPTY
            - FULL
        queries:
          type: array
          description: the list of queries
          items:
            type: string
        tags:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: array
          description: the list of tags
          items:
            type: string
        connector:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: object
          description: the Connector setup bound to a Dataset
          properties:
            id:
              type: string
              x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
              description: the Connector id
            name:
              type: string
              x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
              description: the Connector name
            version:
              type: string
              description: the Connector version
            parametersValues:
              type: object
              additionalProperties:
                type: string
            #      required:
            #       - id
        fragmentsIds:
          type: array
          description: the list of other Datasets ids to compose as fragments
          items:
            type: string
        validatorId:
          type: string
          description: the validator id
        compatibility:
          type: array
          description: the list of compatible Solutions versions
          items:
            $ref: '#/components/schemas/DatasetCompatibility'
        security:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          allOf:
            - $ref: '#/components/schemas/DatasetSecurity'
    #      required:
    #        - name
    #        - connector
    DatasetSecurity:
      type: object
      description: the dataset security information
      properties:
        default:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the role by default
        accessControlList:
          type: array
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the list which can access this Dataset with detailed access control information
          items:
            $ref: '#/components/schemas/DatasetAccessControl'
      required:
        - default
        - accessControlList
    DatasetAccessControl:
      type: object
      description: a Dataset access control item
      properties:
        id:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the identity id
        role:
          type: string
          description: a role
      required:
        - id
        - role
    DatasetRole:
      type: object
      description: the Dataset Role
      properties:
        role:
          type: string
          description: the Dataset Role
      required:
        - role
    DatasetSourceType:
      type: string
      description: the Dataset Source Type
      enum:
        - ADT
        - AzureStorage
        - File
        - None
        - Twincache
    DatasetTwinGraphInfo:
      type: object
      description: a twin graph query in cypher language
      properties:
        jobId:
          type: string
          description: the import job id
        datasetId:
          type: string
          description: the Dataset id
        status:
          type: string
          description: Twingraph status
    DatasetTwinGraphQuery:
      type: object
      description: a twin graph query in cypher language
      properties:
        query:
          type: string
          description: the query in cypher language
      required:
        - query
    DatasetTwinGraphHash:
      type: object
      description: a twin graph hash
      properties:
        hash:
          type: string
          description: the hash of the graph
    DatasetCompatibility:
      type: object
      description: a Dataset compatibility constraint to a Solution version open range
      properties:
        solutionKey:
          type: string
          description: the Solution key which group Solution versions
        minimumVersion:
          type: string
          description: the Solution minimum version compatibility (version included)
        maximumVersion:
          type: string
          description: the Solution maximum version compatibility (version included)
      required:
        - solutionKey
    Validator:
      type: object
      description: a Validator to validate a Dataset
      properties:
        id:
          type: string
          readOnly: true
          description: the Validator id
        name:
          type: string
          description: the Validator name
        description:
          type: string
          description: the Validator description
        repository:
          type: string
          description: the registry repository containing the Validator image
        version:
          type: string
          description: the Validator version MAJOR.MINOR.PATCH. Must be aligned with an existing repository tag
        ownerId:
          type: string
          readOnly: true
          description: the User id which own this Validator
        url:
          type: string
          description: an optional URL link to Validator page
        tags:
          type: array
          description: the list of tags
          items:
            type: string
      required:
        - id
        - name
        - repository
        - version
    ValidatorRun:
      type: object
      description: a Validator Run
      properties:
        id:
          type: string
          readOnly: true
          description: the Validator Run id
        validatorId:
          type: string
          readOnly: true
          description: the Validator id
        validatorName:
          type: string
          readOnly: true
          description: the validator name
        datasetId:
          type: string
          description: the Dataset id to run the validator on
        datasetName:
          type: string
          readOnly: true
          description: the Dataset name
        state:
          type: string
          readOnly: true
          description: the Validator Run state
          enum: ["Running","Finished","OnError"]
        containerId:
          type: string
          readOnly: true
          description: the Validator Run container id
        logs:
          type: string
          readOnly: true
          description: the Validator Run logs
      required:
        - datasetId
    DatasetCopyParameters:
      type: object
      description: the Dataset Copy Parameters
      properties:
        sourceId:
          type: string
          description: the source Dataset id
        targetId:
          type: string
          description: the target Dataset id
        options:
          type: object
          description: freeform options to path to connectors
          additionalProperties: true
    SubDatasetGraphQuery:
      type: object
      description: a twin graph query in cypher language
      properties:
        name:
          type: string
          description: the name of the subdataset
        description:
          type: string
          description: the description of the subdataset
        queries:
          type: array
          description: the query in cypher language
          items:
            type: string
        main:
          type: boolean
          description: is this the main dataset
    SourceInfo:
      type: object
      description: Source job import information
      properties:
        name:
          type: string
          description: the source name containing the files to import
        location:
          type: string
          description: the source location containing the files to import
        path:
          type: string
          description: the source location containing the files to import
        jobId:
          type: string
          description: indicate the last import jobId
      required:
        - location
    TwinGraphBatchResult:
      type: object
      description: Processing result
      properties:
        totalLines:
          type: integer
        processedLines:
          type: integer
        errors:
          type: array
          items:
            type: string
      required:
        - totalLines
        - processedLines
        - errors
    GraphProperties:
      type: object
      properties:
        type:
          type: string
          description: the type of the relationship
        source:
          type: string
          description: the source node of the relationship
        target:
          type: string
          description: the target node of the relationship
        name:
          type: string
          description: the name of the graph data object
        params:
          type: string
          description: the parameters of the graph data object

A Dataset is defined by a name, a type of data source (property sourceType) and a twin graph identifier (property twingraphId). After creating your Dataset you will get an identifier of the form d-xxxxxxxx which will be used to reference it to your scenarios.

Define a Scenario

An example of Scenario.yaml which is a file used by the Cosmo Tech API for our current solution would be :

Scenario.yaml
name: Orchestrator Tutorial
description: Scenario of the Orchestrator tutorial
tags:
  - Brewery
  - Orchestrator Tutorial
runTemplateId: orchestrator_tutorial_2
datasetList:
  - 'd-xxxxxxxx'
parametersValues:
  - parameterId: Stock
    varType: int
    value: 200
  - parameterId: RestockQty
    varType: int
    value: 50
  - parameterId: NbWaiters
    varType: int
    value: 10

Full Open API description of a Scenario is available here

Open API description of a Scenario (components.schemas) 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
    Scenario:
      type: object
      x-class-extra-annotation: "@com.redis.om.spring.annotations.Document"
      description: a Scenario with base information
      properties:
        id:
          type: string
          x-field-extra-annotation: "@org.springframework.data.annotation.Id"
          readOnly: true
          description: the Scenario unique identifier
        name:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          description: the Scenario name
        description:
          type: string
          description: the Scenario description
        tags:
          type: array
          description: the list of tags
          items:
            type: string
        parentId:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the Scenario parent id
        ownerId:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          readOnly: true
          description: the user id which own this Scenario
        rootId:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          readOnly: true
          description: the scenario root id
        solutionId:
          type: string
          readOnly: true
          description: the Solution Id associated with this Scenario
        runTemplateId:
          type: string
          description: the Solution Run Template Id associated with this Scenario
        organizationId:
            type: string
            x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
            readOnly: true
            description: the associated Organization Id
        workspaceId:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          readOnly: true
          description: the associated Workspace Id
        state:
          $ref: '#/components/schemas/ScenarioJobState'
        creationDate:
          type: integer
          format: int64
          readOnly: true
          description: the Scenario creation date
        lastUpdate:
          type: integer
          format: int64
          readOnly: true
          description: the last time a Scenario was updated
        ownerName:
          type: string
          readOnly: true
          description: the name of the owner
        solutionName:
          type: string
          readOnly: true
          description: the Solution name
        runTemplateName:
          type: string
          readOnly: true
          description: the Solution Run Template name associated with this Scenario
        datasetList:
          type: array
          description: the list of Dataset Id associated to this Scenario Run Template
          items:
            type: string
        runSizing:
          description: definition of resources needed for the scenario run
          $ref: "#/components/schemas/ScenarioResourceSizing"
        parametersValues:
          type: array
          description: the list of Solution Run Template parameters values
          items:
            $ref: '#/components/schemas/ScenarioRunTemplateParameterValue'
        lastRun:
          allOf:
            - $ref: '#/components/schemas/ScenarioLastRun'
            - type: object
              description: the last Scenario Run for this Scenario
        parentLastRun:
          allOf:
            - $ref: '#/components/schemas/ScenarioLastRun'
            - type: object
              description: the last Scenario Run for the parent of this Scenario
        rootLastRun:
          allOf:
            - $ref: '#/components/schemas/ScenarioLastRun'
            - type: object
              description: the last Scenario Run for the root (master) of Scenario
        validationStatus:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          allOf:
            - $ref: '#/components/schemas/ScenarioValidationStatus'
        security:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          allOf:
            - $ref: '#/components/schemas/ScenarioSecurity'
#      required:
#        - name
#        - runTemplateId
    ScenarioSecurity:
      type: object
      description: the Scenario security information
      properties:
        default:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the role by default
        accessControlList:
          type: array
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the list which can access this Scenario with detailed access control information
          items:
            $ref: '#/components/schemas/ScenarioAccessControl'
      required:
        - default
        - accessControlList
    ScenarioAccessControl:
      type: object
      description: a Scenario access control item
      properties:
        id:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the identity id
        role:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: a role
      required:
        - id
        - role
    ScenarioRole:
      type: object
      description: the Scenario Role
      properties:
        role:
          type: string
          description: the Scenario Role
      required:
        - role
    ScenarioResourceSizing:
      type: object
      description: a description object for resource requests and limits (default same configuration as basic sizing)
      properties:
        requests:
          $ref: '#/components/schemas/ResourceSizeInfo'
        limits:
          $ref: '#/components/schemas/ResourceSizeInfo'
      required:
        - requests
        - limits
    ResourceSizeInfo:
      type: object
      description: define cpus and memory needs
      properties:
        cpu:
          type: string
          description: define cpu needs
        memory:
          type: string
          description: define memory needs
      required:
        - cpu
        - memory
    ScenarioLastRun:
      type: object
      properties:
        scenarioRunId:
          type: string
          description: the last Scenario Run id
        csmSimulationRun:
          type: string
          description: the last Cosmo Tech Simulation Run id
        workflowId:
          type: string
          description: the last Workflow Id
        workflowName:
          type: string
          description: the last Workflow name
    ScenarioRunTemplateParameterValue:
      type: object
      description: the value of a Solution Run Template parameter for a Scenario
      properties:
        parameterId:
          type: string
          description: the parameter Id
        varType:
          type: string
          readOnly: true
          description: the parameter value type
        value:
          type: string
          description: the parameter value
        isInherited:
          type: boolean
          description: whether or not the value is inherited from parent or has been changed
      required:
        - parameterId
        - value
    ScenarioComparisonResult:
      type: object
      description: the result of the comparison of two scenarios
      properties:
        scenarioId:
          type: string
          readOnly: true
          description: the Scenario Id which is the reference for the comparison
        comparedScenarioId:
          type: string
          readOnly: true
          description: the Scenario Id the reference Scenario is compared to
        changedValues:
          type: array
          readOnly: true
          description: the list of changed values for parameters
          items:
            $ref: '#/components/schemas/ScenarioChangedParameterValue'
    ScenarioChangedParameterValue:
      type: object
      description: the difference between the values of a parameter
      properties:
        parameterId:
          type: string
          readOnly: true
          description: the parameter id the values refer to
        varType:
          type: string
          readOnly: true
          description: the parameter value type
        value:
          type: string
          readOnly: true
          description: the parameter value for the reference Scenario
        comparedValue:
          type: string
          readOnly: true
          description: the parameter value for the compared Scenario
    ScenarioDataDownloadInfo:
      type: object
      description: Scenario data download job info
      properties:
        url:
          type: string
          readOnly: true
          description: the Scenario Data Download URL
        state:
          $ref: '#/components/schemas/ScenarioJobState'
    ScenarioDataDownloadJob:
      type: object
      description: Scenario data download job
      properties:
        id:
          type: string
          readOnly: true
          description: the Scenario Data Download job identifier
    ScenarioJobState:
      type: string
      readOnly: true
      description: the Scenario job state
      enum:
        - Created
        - Running
        - Successful
        - Failed
        # PROD-7888 : When requesting the scenario state right after a run has been submitted,
        # the scenario run service (e.g., Argo Workflow) might not have scheduled the run
        # effectively yet.
        # Furthermore, temporary communication errors might occur anytime when remotely
        # fetching last scenario run statuses.
        - Unknown
        # PROD-7420 : return data ingestion status
        - DataIngestionInProgress
    ScenarioValidationStatus:
      type: string
      description: the validation status of the scenario
      enum:
        - Draft
        - Rejected
        - Unknown
        - Validated

As a result of creating your Scenario you will get an identifier of the form s-xxxxxxxx that you can use to run it (as we will see further) and reference it in other endpoints.

A scenario essentially combines a run template with a dataset. The run template provides a set of parameters that can then be modified by the user before running the scenario. Multiple elements are required for defining a scenario, most notably:

  • runTemplateId: should be one of the run template identifiers defined in our Solution.
  • datasetList: should reference a previously defined Dataset using its identifier.
  • parametersValues: is a list of objects representing the parameters we defined in the Solution, with the actual values they are set to.

Run our distant Scenario

We will need to either use a lot of parameters or environment variables. To make things easier we can make use of a parameter of csm-orc run to generate a template of .env file we will be able to use.

That parameter is --gen-env-target and takes a filename to create a .env file initialized with all the environment variables defined in our run.json

If a variable is set multiple times, only the last one will be taken into account. Each environment variable will have one and only one value from the following list (highest value in the list takes precedence):

  • The value defined in the run.json.
  • The current value in your working environment.
  • The defaultValue defined in the run.json.
  • The description defined in the run.json.
  • None if no other value can be found.
use of --gen-env-target
csm-orc run code/run_templates/orchestrator_tutorial_2/run.json --gen-env-target code/run_templates/orchestrator_tutorial_2/vars.env
cat code/run_templates/orchestrator_tutorial_2/vars.env
# CSM_API_SCOPE=The scope of identification used to request access token for your Cosmo Tech API instance
# CSM_API_URL=The URL used to query your Cosmo Tech API instance
# CSM_DATASET_ABSOLUTE_PATH=Simulation/Resource/scenariorun-data
# CSM_ORGANIZATION_ID=The identifier of the organization in the Cosmo Tech API
# CSM_PARAMETERS_ABSOLUTE_PATH=code/run_templates/orchestrator_tutorial_2
# CSM_SCENARIO_ID=The identifier of the scenario in the Cosmo Tech API
# CSM_SIMULATION=BusinessApp_Simulation
# CSM_WORKSPACE_ID=The id of the workspace in the Cosmo Tech API

This .env file can then be used as a parameter of the docker run command with --env-file or can be used with a tool like dotenv (pip install dotenv) to temporary set the environment variables of commands. You can also just load it to get all values in your current environment.

set -o allexport
source code/run_templates/orchestrator_tutorial_2/vars.env set
+o allexport
csm-orc run code/run_templates/orchestrator_tutorial_2/run.json

docker run --env-file code/run_templates/orchestrator_tutorial_2/vars.env cosmotech/mybrewery_simulator

dotenv -e code/run_templates/orchestrator_tutorial_2/vars.env csm-orc run code/run_templates/orchestrator_tutorial_2/run.json

I want to use my local data instead of needing a Scenario

The csm-orc run command comes with an optional parameter --skip-step that can be set in an environment variable CSM_SKIP_STEPS. This parameter can be given a list of steps that will be ignored during the run.

Since the "only" step used to download distant data from our run is DownloadScenarioData we can simply run locally by skipping it:

Running only the SimulationRun step
csm-orc run --skip-step DownloadScenarioData --skip-step ApplyParameters code/run_templates/orchestrator_tutorial_2/run.json