Skip to content

Make a Cosmo Tech Simulator cloud-ready

In this tutorial we will take our updated Brewery Simulator project as we left it in the previous tutorial and add modification to get it ready to be used inside a Cosmo Tech Platform.

Requirements

  • docker
  • A Cosmo Tech Platform minimal version of 3.0
  • A Cosmo Tech SDK minimal version of 10.0
  • Installed SDK package must include csmcli

How does the Cosmo Tech Platform runs orchestration?

The Cosmo Tech Platform will run pre-registered docker images based on the definition of Solutions in its API.

In the API you can define a Solution that will refer to a specific version of a docker image in a given repository. That Solution defines sets of Parameters and Run Templates. Then Workspaces are defined in the API and point to a given Solution allowing you to create a Scenario which will tie together Datasets, Parameters, and a Run Template which will then be run by the API.

Datasets are a description of how and where data are stored, Parameters can be simple types or full Datasets allowing you to update the main Dataset of your Scenario, therefore changing its initial state or behavior (or both).

Running your Scenario will create a ScenarioRun into the API and that ScenarioRun will make use of the docker image your previously defined. The API will call the image with an Entrypoint called entrypoint.py and that call should be your full orchestration.

Why entrypoint.py and not csm-orc ?

The definition of the Entrypoint being entrypoint.py is left for compatibility with old images on the Cosmo Tech API (pre 3.0). It will be changed in the future, but for now csm-orc comes bundled with a script called entrypoint.py that allows both legacy and new runs of the docker image.

How does entrypoint.py works?

The entrypoint.py command (and its equivalent csm-orc entrypoint) works using some Environment variables to switch between 3 modes:

  • The general entrypoint with no special configuration that allow to run your run templates requiring the environment variable CSM_RUN_TEMPLATE_ID to be set with your run template name.
  • The legacy entrypoint activated by setting the environment variable CSM_ENTRYPOINT_LEGACY to true which is made available for compatibility with pre 3.0 Cosmo Tech API.
  • The direct simulator mode which requires the environment variable CSM_RUN_TEMPLATE_ID to not be set, and will then run the simulator by passing it any argument of the command.

Every Environment Variable passed to the command will be forwarded to the csm-orc run command inside, and the command will be run with the working directory set to /pkg/share.

Which Environment Variables are made available by the API?

The Cosmo Tech API will forward a set of environment variables to any Simulator containers. You can find the full list in the following table.

List of environment variables
Environment Variable Description
AZURE_CLIENT_ID An identifier to an Azure identity defined during the installation of the platform
AZURE_CLIENT_SECRET A secret tied to the given client ID allowing to connect with it
AZURE_TENANT_ID An identifier of your Azure tenant to be able to connect to it
IDENTITY_PROVIDER Will be set to azure in an Azure based API
CSM_API_URL The url to the Cosmo Tech API your are connecting to
CSM_API_SCOPE An identifier scope used to get permission to your API
CSM_DATASET_ABSOLUTE_PATH A local folder path to which an external volume is mounted to and in which you can write datasets
CSM_PARAMETERS_ABSOLUTE_PATH A local folder path to which an external volume is mounted to and in which you can write parameters
TWIN_CACHE_HOST The URL to the twin cache service inside the platform (deprecated)
TWIN_CACHE_PORT The port to the twin cache service inside the platform (deprecated)
TWIN_CACHE_PASSWORD The password to the twin cache service inside the platform (deprecated)
TWIN_CACHE_USERNAME The username to the twin cache service inside the platform (deprecated)
CSM_SIMULATION_ID The Simulation ID for your current scenario run
AZURE_DATA_EXPLORER_RESOURCE_URI The URI used to query an Azure Data Explorer Cluster tied to your Workspace
AZURE_DATA_EXPLORER_RESOURCE_INGEST_URI The URI used to send data to an Azure Data Explorer Cluster tied to your Workspace
AZURE_DATA_EXPLORER_DATABASE_NAME The name of the database of your Workspace in Azure Data Explorer
CSM_ORGANIZATION_ID The ID of your Organization in the Cosmo Tech API
CSM_WORKSPACE_ID The ID of your Workspace in the Cosmo Tech API
CSM_SCENARIO_ID The ID of your Scenario in the Cosmo Tech API
CSM_RUN_TEMPLATE_ID The ID of your Run Template in the Cosmo Tech API (as defined in your Solution)
CSM_CONTAINER_MODE Will always be csm-orc left for compatibility with pre 3.0 API
CSM_ENTRYPOINT_LEGACY Will always be false
CSM_PROBES_MEASURES_TOPIC An amqp address used by the simulator to send your probe data to ADX via Event Hub
CSM_CONTROL_PLANE_TOPIC An amqp address used by the simulator to send your control info to ADX via Event Hub (mostly number of measures sent)
CSM_SIMULATION The Simulation file that should be used to run your simulation (this value is defined in your Run Template and targets one of the files in the Simulation/ folder)

Connect to the API to get our Scenario data

Multiple ways exists to connect to the API and query some data, but for simplicity there exists a command in csm-orc that will make use of known API environment variables and do the work for you.

That command is csm-orc fetch-scenariorun-data (documentation of the command is available here).

The command makes use of 5 environment variables set by the API (as described in the previous section):

  • CSM_ORGANIZATION_ID
  • CSM_WORKSPACE_ID
  • CSM_SCENARIO_ID
  • CSM_DATASET_ABSOLUTE_PATH
  • CSM_PARAMETERS_ABSOLUTE_PATH

And uses 3 control environment variables:

  • WRITE_JSON: If set to true will write a parameters.json file in CSM_PARAMETERS_ABSOLUTE_PATH.
  • WRITE_CSV: If set to true will write a parameters.csv file in CSM_PARAMETERS_ABSOLUTE_PATH.
  • FETCH_DATASET: If set to true will download all the Datasets tied to your scenario, will write the main ones (not defined as a Parameter) in CSM_DATASET_ABSOLUTE_PATH and will write the others in CSM_PARAMETERS_ABSOLUTE_PATH/[parameter name] where [parameter name] is the name of the Parameter targeting the Dataset (with a type set to %DATASETID%).

With that command one can easily download any dataset defined in the Cosmo Tech API, as long as those datasets use one of the standard connections defined in the API (as of version 3.0: Azure Blob Storage, Azure Digital Twin, TwinGraph Storage).

Combine everything in a Run Template

Using all those new information we can see that most of the actions needed to run our code based on API data are already prepackaged.

We can:

  • Download our scenario information using csm-orc fetch-scenariorun-data.
  • Apply our parameters with our apply_parameters.py.
  • Run our simulation using csm-orc run-step.
  • And send our simulation results to an external system (here Azure Data Explorer) by setting environment variables during the run-step.

It is then easy to update our previous run.json to take those changes into account.

New code/run_templates/orchestrator_tutorial_2 folder

To keep a clean distinction on the code of the previous tutorial and this one, we will refer to the folder code/run_templates/orchestrator_tutorial_2 which is an exact copy of the folder code/run_templates/orchestrator_tutorial_1.

You can create it by using the following command:

copy `orchestrator_tutorial_1`
cp -r code/run_templates/orchestrator_tutorial_1 code/run_templates/orchestrator_tutorial_2

code/run_templates/orchestrator_tutorial2/run.json
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
{
  "steps": [
    {
      "id": "DownloadScenarioData",
      "command": "csm-orc",
      "description": "Connect to the Cosmo Tech API to download dataset and parameters",
      "arguments": [
        "fetch-scenariorun-data",
        "--write-json"
      ],
      "useSystemEnvironment": true,
      "environment": {
        "CSM_API_URL": {
          "description": "The URL used to query your Cosmo Tech API instance"
        },
        "CSM_API_SCOPE": {
          "description": "The scope of identification used to request access token for your Cosmo Tech API instance"
        },
        "CSM_ORGANIZATION_ID": {
          "description": "The id of the organization in the Cosmo Tech API"
        },
        "CSM_WORKSPACE_ID": {
          "description": "The id of the workspace in the Cosmo Tech API"
        },
        "CSM_SCENARIO_ID": {
          "description": "The id of the scenario in the Cosmo Tech API"
        },
        "CSM_DATASET_ABSOLUTE_PATH": {
          "description": "The path used to store the datasets of the scenario"
        },
        "CSM_PARAMETERS_ABSOLUTE_PATH": {
          "description": "The path used to store the parameters of the scenario"
        }
      }
    },
    {
      "id": "ApplyParameters",
      "command": "python",
      "arguments": [
        "code/run_templates/orchestrator_tutorial_2/apply_parameters.py",
        "$CSM_DATASET_ABSOLUTE_PATH",
        "$CSM_DATASET_ABSOLUTE_PATH",
        "$CSM_PARAMETERS_ABSOLUTE_PATH/parameters.json"
      ],
      "description": "Apply our parameters to the dataset",
      "environment": {
        "CSM_DATASET_ABSOLUTE_PATH": {
          "description": "The path to the dataset to update",
          "defaultValue": "Simulation/Resource/scenariorun-data"
        },
        "CSM_PARAMETERS_ABSOLUTE_PATH": {
          "description": "The path to a folder containing our parameters.json file",
          "defaultValue": "code/run_templates/orchestrator_tutorial_2"
        }
      },
      "precedents": [
        "DownloadScenarioData"
      ]
    },
    {
      "id": "SimulationRun",
      "command": "csm-orc",
      "arguments": [
        "run-step",
        "--template",
        "orchestrator_tutorial_2",
        "--steps",
        "engine"
      ],
      "description": "Runs the simulation targeted by CSM_SIMULATION",
      "useSystemEnvironment": true,
      "environment": {
        "CSM_SIMULATION": {
          "description": "The simulation file to run",
          "defaultValue": "BusinessApp_Simulation"
        }
      },
      "precedents": [
        "ApplyParameters"
      ]
    }
  ]
}

We can see a few changes and additions compared to the previous run.json file:

  • We created a new step called DownloadScenarioData that makes use of csm-orc fetch-scenariorun-data:
    • In this step we defined a few environment variables required to run it, but we still added useSystemEnvironment to true to ensure any environment variable required to connect to the API is made available as well.
  • In the ApplyParameters we made a few changes:
    • DATASET/PARAMETERS_PATH became CSM_DATASET/PARAMETERS_ABSOLUTE_PATH.
    • We added /parameters.json in the arguments for the parameters path.
    • We added a precendent step to schedule it after the DownloadScenarioData.
  • In the SimulatorRun step changes are minimal:
    • The template targeted is the new orchestrator_tutorial_2.
    • The default value of CSM_SIMULATION has been changed to BusinessApp_Simulation since docker simulation can't make use of visual consumers.

Build a docker image

Now that we have our run.json ready we can build our docker image.

Build a docker image using csmcli
# First we will clean everything to ensure a correct generation of the project
csm clean
# Now we can build our full project
csm flow
# The simulator is now ready to be packaged
csm docker build

After running those commands we end up with a docker image cosmotech/mybrewery_simulator ready to be used.

A simple test can be made with the following command:

Test the docker image
docker run -e CSM_RUN_TEMPLATE_ID=orchestrator_tutorial_1 -e CSM_SIMULATION=BusinessApp_Simulation -v ./Simulation/Resource/scenariorun-data:/mnt/scenariorun-data cosmotech/mybrewery_simulator

About the parameters of the command docker run

  • -e sets an environment variable during the docker run, here we set our run template id and our simulation.
  • -v sets a volume during the docker run, here we mount our local Simulation/Resource/scenariorun-data folder to the path /mnt/scenariorun-data in the docker container, allowing us to use our existing dataset during the run.

Define API items

The following sections give you simple examples of .yaml files used to define resources in the Cosmo Tech API.

Define a Solution

An example of Solution.yaml (a file used by the Cosmo Tech API) for our current solution would be:

Solution.yaml
key: "brewery"
name: "Demo Brewery Solution, csm-orc tutorial"
description: "A simple example of a Cosmo Tech solution built using the Brewery model"
tags:
  - "csm-orc"
  - "Brewery"
repository: "mybrewery_simulator"
version: "0.0.1"
parameters:
  - id: "Stock"
    varType: "int"
    defaultValue: 100
  - id: "RestockQty"
    varType: "int"
    defaultValue: 25
  - id: "NbWaiters"
    varType: "int"
    defaultValue: 5
parameterGroups:
  - id: bar_parameters
    parameters:
      - Stock
      - RestockQty
      - NbWaiters
runTemplates:
  - id: "orchestrator_tutorial_2"
    name: "Basic what-if"
    description: "Basic what-if simulation"
    parameterGroups:
      - bar_parameters
    csmSimulation: BusinessApp_Simulation

Full Open API description of a Solution is available here.

Open API description of a Solution (components.schemas) 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
        description: the Organization identifier
        required: true
        schema:
          type: string
      - name: solution_id
        in: path
        description: the Solution identifier
        required: true
        schema:
          type: string
      - name: identity_id
        in: path
        description: the User identifier
        required: true
        schema:
          type: string
    get:
      operationId: getSolutionAccessControl
      tags:
        - solution
      summary: Get a control access for the Solution
      responses:
        "200":
          description: The Solution access
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SolutionAccessControl'
              examples:
                SolutionAccessControl:
                  $ref: '#/components/examples/BrewerySolutionAccessControl'
        "404":
          description: The Solution or user specified is unknown or you don't have access to it
    patch:
      operationId: updateSolutionAccessControl
      tags:
        - solution
      summary: Update the specified access to User for a Solution
      requestBody:
        description: The new Solution Access Control
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/SolutionRole'
            examples:
              BrewerySolutionRole:
                $ref: '#/components/examples/BrewerySolutionRole'
      responses:
        "200":
          description: The Solution access
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SolutionAccessControl'
              examples:
                SolutionAccessControl:
                  $ref: '#/components/examples/BrewerySolutionAccessControl'
        "404":
          description: The Solution specified is unknown or you don't have access to it
    delete:
      operationId: removeSolutionAccessControl
      tags:
        - solution
      summary: Remove the specified access from the given Organization Solution
      responses:
        "204":
          description: Request succeeded
        "404":
          description: The Solution or the user specified is unknown or you don't have access to them

components:
  securitySchemes:
    oAuth2AuthCode:
      type: oauth2
      description: OAuth2 authentication
      flows:
        implicit:
          authorizationUrl: https://login.microsoftonline.com/common/oauth2/v2.0/authorize
          scopes:
            http://dev.api.cosmotech.com/platform: Platform scope
  schemas:
    Solution:
      type: object
      x-class-extra-annotation: "@com.redis.om.spring.annotations.Document"
      description: a version of a Solution
      properties:
        id:
          type: string
          x-field-extra-annotation: "@org.springframework.data.annotation.Id"
          readOnly: true
          description: the Solution version unique identifier
        organizationId:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: string
          readOnly: true
          description: the Organization unique identifier
        key:
          type: string
          description: the Solution key which group Solution versions
        name:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          description: the Solution name
        description:
          type: string
          description: the Solution description
        repository:
          type: string
          description: the registry repository containing the image
        alwaysPull:
          type: boolean
          description: set to true if the runtemplate wants to always pull the image
          default: false
        csmSimulator:
          type: string
          description: the main Cosmo Tech simulator name used in standard Run Template
        version:
          type: string
          description: the Solution version MAJOR.MINOR.PATCH. Must be aligned with an existing repository tag
        ownerId:
          type: string
          readOnly: true
          description: the User id which own this Solution
        sdkVersion:
          type: string
          description: the MAJOR.MINOR version used to build this solution
        url:
          type: string
          description: an optional URL link to solution page
        tags:
          type: array
          description: the list of tags
          items:
            type: string
        parameters:
          type: array
          description: the list of Run Template Parameters
          items:
            $ref: '#/components/schemas/RunTemplateParameter'
        parameterGroups:
          type: array
          description: the list of parameters groups for the Run Templates
          items:
            $ref: '#/components/schemas/RunTemplateParameterGroup'
        runTemplates:
          type: array
          description: list of Run Template
          items:
            $ref: '#/components/schemas/RunTemplate'
        security:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          allOf:
            - $ref: '#/components/schemas/SolutionSecurity'
  #       required:
  #         - key
  #         - name
  #         - repository
  #         - version
  #         - runTemplates
    SolutionSecurity:
      type: object
      description: the Solution security information
      properties:
        default:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the role by default
        accessControlList:
          type: array
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the list which can access this Solution with detailed access control information
          items:
            $ref: '#/components/schemas/SolutionAccessControl'
      required:
        - default
        - accessControlList
    SolutionAccessControl:
      type: object
      description: a Solution access control item
      properties:
        id:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the identity id
        role:
          type: string
          description: a role
      required:
        - id
        - role
    SolutionRole:
      type: object
      description: the Solution Role
      properties:
        role:
          type: string
          description: the Solution Role
      required:
        - role
    RunTemplate:
      type: object
      description: a Solution Run Template
      properties:
        id:
          type: string
          description: the Solution Run Template id
        name:
          type: string
          description: the Run Template name
          deprecated: true
        labels:
          $ref: '#/components/schemas/TranslatedLabels'
        description:
          type: string
          description: the Run Template description
        csmSimulation:
          type: string
          description: the Cosmo Tech simulation name. This information is send to the Engine. Mandatory information if no Engine is defined
        tags:
          type: array
          description: the list of Run Template tags
          items:
            type: string
        computeSize:
          type: string
          description: the compute size needed for this Run Template. Standard sizes are basic and highcpu. Default is basic
        runSizing:
          description: definition of resources needed for the scenario run
          $ref: "#/components/schemas/RunTemplateResourceSizing"
        noDataIngestionState:
          type: boolean
          description: set to true if the run template does not want to check data ingestion state (no probes or not control plane)
        fetchDatasets:
          type: boolean
          description: whether or not the fetch dataset step is done
        scenarioDataDownloadTransform:
          type: boolean
          description: whether or not the scenario data download transform step step is done
        fetchScenarioParameters:
          type: boolean
          description: whether or not the fetch parameters step is done
        applyParameters:
          type: boolean
          description: whether or not the apply parameter step is done
        validateData:
          type: boolean
          description: whether or not the validate step is done
        sendDatasetsToDataWarehouse:
          type: boolean
          description: whether or not the Datasets values are send to the DataWarehouse prior to Simulation Run. If not set follow the Workspace setting
        sendInputParametersToDataWarehouse:
          type: boolean
          description: whether or not the input parameters values are send to the DataWarehouse prior to Simulation Run. If not set follow the Workspace setting
        preRun:
          type: boolean
          description: whether or not the pre-run step is done
        run:
          type: boolean
          description: whether or not the run step is done
        postRun:
          type: boolean
          description: whether or not the post-run step is done
        parametersJson:
          type: boolean
          description: whether or not to store the scenario parameters in json instead of csv
        parametersHandlerSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        datasetValidatorSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        preRunSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        runSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        postRunSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        scenariodataTransformSource:
          $ref: "#/components/schemas/RunTemplateStepSource"
        parameterGroups:
          type: array
          description: the ordered list of parameters groups for the Run Template
          items:
            type: string
            description: a Run Template Group Parameter id
        stackSteps:
          type: boolean
          description: whether or not to stack adjacent scenario run steps in one container run which will chain steps
        gitRepositoryUrl:
          type: string
          description: an optional URL to the git repository
        gitBranchName:
          type: string
          description: an optional git branch name
        runTemplateSourceDir:
          type: string
          description: an optional directory where to find the run template source
        orchestratorType:
          $ref: "#/components/schemas/RunTemplateOrchestrator"
        executionTimeout:
          type: integer
          description: an optional duration in seconds in which a workflow is allowed to run
        deleteHistoricalData:
          $ref: "#/components/schemas/DeleteHistoricalData"
      required:
        - id
#         - name
    DeleteHistoricalData:
      type: object
      description: Configuration of scenario runs deletion automatic mecanism
      properties:
        enable:
          type: boolean
          description: Activate or Deactivate historical data deletion
          default: true
        pollFrequency:
          type: integer
          description: define the polling frequency of scenario run status (in millis)
          default: 10000
        timeOut:
          type: integer
          description: define the polling timeout
          default: 28800
      required:
        - enable

    RunTemplateOrchestrator:
      type: string
      description: the Orchestrator to use for the Run Template
      enum:
        - argoWorkflow
        - csmOrc
    RunTemplateHandlerId:
      type: string
      description: the Run Template step handler identifier
      enum: ["parameters_handler","validator","prerun","engine","postrun","scenariodata_transform"]
    RunTemplateStepSource:
      type: string
      description: the source local, cloud or git repository
      enum: ["cloud","git","local","platform"]
    RunTemplateResourceSizing:

Once this file is sent to a Cosmo Tech API we will get a Solution ID of the form sol-xxxxxxxx which will allow us to create further elements inside the API referencing our solution.

Now let's look deeper at how to create our Solution.yaml.

Solution description

Solution.yaml - description
key: "brewery"
name: "Demo Brewery Solution, csm-orc tutorial"
description: "A simple example of a Cosmo Tech solution built using the Brewery model"
tags:
  - "csm-orc"
  - "Brewery"
repository: "mybrewery_simulator"
version: "0.0.1"

We can see a few important properties in this part of the Solution file:

  • key: it is a "grouping" value used to make multiple Solution using the same base but different versions together.
  • name: a simple name given to the Solution.
  • description: a simple description of the Solution.
  • tags: a list of tags used in the API to filter Solution resources.
  • repository: the name of the docker image inside the image registry.
  • version: the version of the docker image tied to the Solution.

parameters and parameterGroups

Solution.yaml - parameters
parameters:
  - id: "Stock"
    varType: "int"
    defaultValue: 100
  - id: "RestockQty"
    varType: "int"
    defaultValue: 25
  - id: "NbWaiters"
    varType: "int"
    defaultValue: 5
parameterGroups:
  - id: bar_parameters
    parameters:
      - Stock
      - RestockQty
      - NbWaiters

In this part we declare our scenario parameters. As previously decided we need three parameters:

  • NbWaiters
  • RestockQty
  • Stock

Each parameter can be given a type (property varType) and a default value (property 'defaultValue'). Then we grouped our parameters in a parametersGroup "bar_parameters". Parameter groups will make our parameters available in further part of the Solution.

runTemplates

Solution.yaml - runTemplates
runTemplates:
  - id: "orchestrator_tutorial_2"
    name: "Basic what-if"
    description: "Basic what-if simulation"
    parameterGroups:
      - bar_parameters
    csmSimulation: BusinessApp_Simulation

In this part we finally define our Run Template for the API. We have three main elements to define:

  • id: the id of a run template is the name of a folder inside code/run_templates that will contain a run.json file, along with one sub-folder for each step in the run template.
  • parameterGroups: list of parameter groups available for the run template. Parameters declared in those groups can be changed by users in order to create new scenarios.
  • csmSimulation: the name of a simulation file (without the .sml.xml suffix), to be set in the CSM_SIMULATION_ID environment variable. This simulation will run as part of the run template execution.

Define a Workspace

An example of Workspace.yaml which is a file used by the Cosmo Tech API for our current solution would be:

Workspace.yaml
key: brewery_orchestrator_tutorial
name: Brewery tutorial workspace
solution:
  solutionId: "sol-xxxxxxxx"

Full Open API description of a Workspace is available here.

Open API description of a Workspace (components.schemas) 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
        in: path
        description: the Workspace identifier
        required: true
        schema:
          type: string
    get:
      operationId: getWorkspaceSecurityUsers
      tags:
        - workspace
      summary: Get the Workspace security users list
      responses:
        "200":
          description: The Workspace security users list
          content:
            application/json:
              schema:
                type: array
                items:
                  type: string
                example:
                  - alice@mycompany.com
                  - bob@mycompany.com
        "404":
          description: the Workspace or the User specified is unknown or you don't have access to them
  /organizations/{organization_id}/workspaces/{workspace_id}/secret:
    parameters:
      - name: organization_id
        in: path
        description: the Organization identifier
        required: true
        schema:
          type: string
      - name: workspace_id
        in: path
        description: the Workspace identifier
        required: true
        schema:
          type: string
    post:
      operationId: createSecret
      tags:
        - workspace
      summary: Create a secret for the Workspace
      requestBody:
        description: the definition of the secret
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/WorkspaceSecret'
            examples:
              Workspace:
                $ref: '#/components/examples/WorkspaceSecretExample'
          application/yaml:
            schema:
              type: string
              format: binary
            examples:
              Workspace:
                $ref: '#/components/examples/WorkspaceSecretExample'
      responses:
        "201":
          description: Request succeeded
        "404":
          description: the Workspace or the User specified is unknown or you don't have access to them


components:
  securitySchemes:
    oAuth2AuthCode:
      type: oauth2
      description: OAuth2 authentication
      flows:
        implicit:
          authorizationUrl: https://login.microsoftonline.com/common/oauth2/v2.0/authorize
          scopes:
            http://dev.api.cosmotech.com/platform: Platform scope
  schemas:
    Workspace:
      type: object
      x-class-extra-annotation: "@com.redis.om.spring.annotations.Document"
      description: a Workspace
      properties:
        id:
          type: string
          x-field-extra-annotation: "@org.springframework.data.annotation.Id"
          readOnly: true
          description: Workspace unique identifier generated by the API
          example: "w-0123456789ab"
        organizationId:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: string
          readOnly: true
          description: Organization unique identifier under which the workspace resides
          example: "o-0123456789ab"
        key:
          type: string
          description: technical key for resource name convention and version grouping. Must be unique
          example: "MyKey"
        name:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          description: Workspace name. This name is display in the sample webApp
          example: "FranceOffice"
        description:
          type: string
          description: the Workspace description
        linkedDatasetIdList:
          type: array
          description: list of dataset linked to this dataset
          items:
            type: string
        version:
          type: string
          description: the Workspace version MAJOR.MINOR.PATCH.
          example: "1.0.0"
        tags:
          type: array
          description: the list of tags
          items:
            type: string
        ownerId:
          type: string
          readOnly: true
          description: the user id which own this workspace. set by the API from Authentification service receved id
        solution:
          $ref: '#/components/schemas/WorkspaceSolution'
        webApp:
          $ref: '#/components/schemas/WorkspaceWebApp'
        sendInputToDataWarehouse:
          type: boolean
          description: default setting for all Scenarios and Run Templates to set whether or not the Dataset values and the input parameters values are send to the DataWarehouse prior to the ScenarioRun
        useDedicatedEventHubNamespace:
          type: boolean
          default: false
          description: Set this property to true to use a dedicated Azure Event Hub Namespace for this Workspace.
            The Event Hub Namespace must be named \'<organization_id\>-<workspace_id\>\' (in lower case).
            This Namespace must also contain two Event Hubs named \'probesmeasures\' and \'scenariorun\'.
        dedicatedEventHubSasKeyName:
          type: string
          description: the Dedicated Event Hub SAS key name, default to RootManageSharedAccessKey. Use the /secret endpoint to set the key value
        dedicatedEventHubAuthenticationStrategy:
          type: string
          description: the Event Hub authentication strategy, SHARED_ACCESS_POLICY or TENANT_CLIENT_CREDENTIALS. Default to the one defined for the tenant.
        sendScenarioRunToEventHub:
          type: boolean
          default : true
          description: default setting for all Scenarios and Run Templates to set whether or not the ScenarioRun is send to the Event Hub
        sendScenarioMetadataToEventHub:
          type: boolean
          default: false
          description: Set this property to false to not send scenario metada to Azure Event Hub Namespace for this Workspace.
            The Event Hub Namespace must be named \'<organization_id\>-<workspace_id\>\' (in lower case).
            This Namespace must also contain two Event Hubs named \'scenariometadata\' and \'scenariorunmetadata\'.
        datasetCopy:
          type: boolean
          default: true
          description: Activate the copy of dataset on scenario creation, meaning that each scenario created in this workspace
            will make this copy. when false, scenario use directly the dataset specified.
        security:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          allOf:
           - $ref: '#/components/schemas/WorkspaceSecurity'

After creating a Workspace resource using the API you will need to create additional resources for it in your platform if you want to run scenarios.

Once your Workspace is created you will get a workspace identifier of the form w-xxxxxxxx. It will allow you to connect to your workspace and reference it in other resources.

We have 3 required parts in a Workspace:

  • key: a unique identifier for your workspace which will be used during resource creation.
  • name: the name of your Workspace.
  • solution.solutionId: the solution identifier (sol-xxxxxxxx) for the solution you want to use in your workspace.

Define a Dataset

An example of Dataset.yaml which is a file used by the Cosmo Tech API for our current solution would be:

Dataset.yaml
name: Orchestrator Tutorial Dataset
description: Example dataset defined during the Orchestrator Tutorial
tags:
  - Brewery
  - Tutorial

Full Open API description of a Dataset is available here.

Open API description of a Dataset (components.schemas) 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
        description: the new Dataset security access to add.
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DatasetAccessControl'
            examples:
              DatasetAccess:
                $ref: '#/components/examples/BreweryDatasetAccessControl'
          application/yaml:
            schema:
              type: string
              format: binary
            examples:
              DatasetAccess:
                $ref: '#/components/examples/BreweryDatasetAccessControl'
      responses:
        "201":
          description: The Dataset access
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DatasetAccessControl'
              examples:
                DatasetAccessControl:
                  $ref: '#/components/examples/BreweryDatasetAccessControl'
        "404":
          description: the Dataset specified is unknown or you don't have access to it
  /organizations/{organization_id}/datasets/{dataset_id}/security/access/{identity_id}:
    parameters:
      - name: organization_id
        in: path
        description: the Organization identifier
        required: true
        schema:
          type: string
      - name: dataset_id
        in: path
        description: the Dataset identifier
        required: true
        schema:
          type: string
      - name: identity_id
        in: path
        description: the User identifier
        required: true
        schema:
          type: string
    get:
      operationId: getDatasetAccessControl
      tags:
        - dataset
      summary: Get a control access for the Dataset
      responses:
        "200":
          description: The Dataset access
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DatasetAccessControl'
              examples:
                DatasetAccessControl:
                  $ref: '#/components/examples/BreweryDatasetAccessControl'
        "404":
          description: The Dataset or user specified is unknown or you don't have access to it
    patch:
      operationId: updateDatasetAccessControl
      tags:
        - dataset
      summary: Update the specified access to User for a Dataset
      requestBody:
        description: The new Dataset Access Control
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DatasetRole'
            examples:
              BreweryDatasetRole:
                $ref: '#/components/examples/BreweryDatasetRole'
      responses:
        "200":
          description: The Dataset access
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DatasetAccessControl'
              examples:
                DatasetAccessControl:
                  $ref: '#/components/examples/BreweryDatasetAccessControl'
        "404":
          description: The Dataset specified is unknown or you don't have access to it
    delete:
      operationId: removeDatasetAccessControl
      tags:
        - dataset
      summary: Remove the specified access from the given Dataset
      responses:
        "204":
          description: Request succeeded
        "404":
          description: The Dataset or the user specified is unknown or you don't have access to them

  /organizations/{organization_id}/datasets/{dataset_id}/security/users:
    parameters:
      - name: organization_id
        in: path
        description: the Organization identifier
        required: true
        schema:
          type: string
      - name: dataset_id
        in: path
        description: the Dataset identifier
        required: true
        schema:
          type: string
    get:
      operationId: getDatasetSecurityUsers
      tags:
        - dataset
      summary: Get the Dataset security users list
      responses:
        "200":
          description: The Dataset security users list
          content:
            application/json:
              schema:
                type: array
                items:
                  type: string
                example:
                  - alice@mycompany.com
                  - bob@mycompany.com
        "404":
          description: the Dataset or the User specified is unknown or you don't have access to them

  /organizations/{organization_id}/datasets/{dataset_id}/link:
    parameters:
      - name: organization_id
        in: path
        description: the Organization identifier
        required: true
        schema:
          type: string
      - name: dataset_id
        in: path
        description: the Dataset identifier
        required: true
        schema:
          type: string
    post:
      parameters:
        - name: workspaceId
          in: query
          description: workspace id to be linked to
          required: true
          schema:
            type: string
      operationId: linkWorkspace
      tags:
        - dataset
      responses:
        "200":
          description: the dataset details
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Dataset'
              examples:
                BreweryADTUpdated:
                  $ref: '#/components/examples/BreweryADTWithLink'
        "400":
          description: Bad request
        "404":
          description: the Dataset specified is unknown or you don't have access to it

  /organizations/{organization_id}/datasets/{dataset_id}/unlink:
    parameters:
      - name: organization_id
        in: path
        description: the Organization identifier
        required: true
        schema:
          type: string
      - name: dataset_id
        in: path
        description: the Dataset identifier
        required: true
        schema:
          type: string
    post:
      parameters:
        - name: workspaceId
          in: query
          description: workspace id to be linked to
          required: true
          schema:
            type: string
      operationId: unlinkWorkspace
      tags:
        - dataset
      responses:
        "200":
          description: the dataset details
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Dataset'
              examples:
                BreweryADT:
                  $ref: '#/components/examples/BreweryADT'
        "400":
          description: Bad request
        "404":
          description: the Dataset specified is unknown or you don't have access to it

components:
  securitySchemes:
    oAuth2AuthCode:
      type: oauth2
      description: OAuth2 authentication
      flows:
        implicit:
          authorizationUrl: https://login.microsoftonline.com/common/oauth2/v2.0/authorize
          scopes:
            http://dev.api.cosmotech.com/platform: Platform scope
  schemas:
    FileUploadValidation:
      type: object
      description: files read on upload
      properties:
        nodes:
          type: array
          description: list of filename found on nodes folder
          items:
            $ref: '#/components/schemas/FileUploadMetadata'
        edges:
          type: array
          description: list of filename found on edges folder
          items:
            $ref: '#/components/schemas/FileUploadMetadata'
    FileUploadMetadata:
      type: object
      properties:
        name:
          type: string
        size:
          type: integer
    DatasetSearch:
      type: object
      description: the search options
      properties:
        datasetTags:
          type: array
          description: the dataset tag list to search
          items:
            type: string
      required:
        - datasetTags
    Dataset:
      type: object
      x-class-extra-annotation: "@com.redis.om.spring.annotations.Document"
      description: a Dataset
      properties:
        id:
          x-field-extra-annotation: "@org.springframework.data.annotation.Id"
          type: string
          readOnly: true
          description: the Dataset unique identifier
        name:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          type: string
          description: the Dataset name
        description:
          type: string
          description: the Dataset description
        ownerId:
          type: string
          readOnly: true
          description: the User id which own this Dataset
        ownerName:
          type: string
          readOnly: true
          description: the name of the owner
        organizationId:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: string
          readOnly: true
          description: the Organization Id related to this Dataset
        parentId:
          type: string
          description: the Dataset id which is the parent of this Dataset
        linkedWorkspaceIdList:
          type: array
          description: list of workspace linked to this dataset
          items:
            type: string
        twingraphId:
          type: string
          description: the twin graph id
        main:
          type: boolean
          description: is this the main dataset
        creationDate:
          type: integer
          format: int64
          readOnly: true
          description: the Dataset creation date
        refreshDate:
          type: integer
          format: int64
          readOnly: true
          description: the last time a refresh was done
        sourceType:
          $ref: '#/components/schemas/DatasetSourceType'
        source:
          $ref: '#/components/schemas/SourceInfo'
        ingestionStatus:
          type: string
          description: the Dataset ingestion status
          nullable: true
          enum:
            - NONE
            - PENDING
            - ERROR
            - SUCCESS
        twincacheStatus:
          type: string
          description: the twincache data status
          nullable: true
          enum:
            - EMPTY
            - FULL
        queries:
          type: array
          description: the list of queries
          items:
            type: string
        tags:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: array
          description: the list of tags
          items:
            type: string
        connector:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          type: object
          description: the Connector setup bound to a Dataset
          properties:
            id:
              type: string
              x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
              description: the Connector id
            name:
              type: string
              x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
              description: the Connector name
            version:
              type: string
              description: the Connector version
            parametersValues:
              type: object
              additionalProperties:
                type: string
            #      required:
            #       - id
        fragmentsIds:
          type: array
          description: the list of other Datasets ids to compose as fragments

A Dataset is defined by a name, a type of data source (property sourceType) and a twin graph identifier (property twingraphId). After creating your Dataset you will get an identifier of the form d-xxxxxxxx which will be used to reference it to your scenarios.

Define a Scenario

An example of Scenario.yaml which is a file used by the Cosmo Tech API for our current solution would be :

Scenario.yaml
name: Orchestrator Tutorial
description: Scenario of the Orchestrator tutorial
tags:
  - Brewery
  - Orchestrator Tutorial
runTemplateId: orchestrator_tutorial_2
datasetList:
  - 'd-xxxxxxxx'
parametersValues:
  - parameterId: Stock
    varType: int
    value: 200
  - parameterId: RestockQty
    varType: int
    value: 50
  - parameterId: NbWaiters
    varType: int
    value: 10

Full Open API description of a Scenario is available here

Open API description of a Scenario (components.schemas) 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
            http://dev.api.cosmotech.com/platform: Platform scope
  schemas:
    Scenario:
      type: object
      x-class-extra-annotation: "@com.redis.om.spring.annotations.Document"
      description: a Scenario with base information
      properties:
        id:
          type: string
          x-field-extra-annotation: "@org.springframework.data.annotation.Id"
          readOnly: true
          description: the Scenario unique identifier
        name:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          description: the Scenario name
        description:
          type: string
          description: the Scenario description
        tags:
          type: array
          description: the list of tags
          items:
            type: string
        parentId:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the Scenario parent id
        ownerId:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          readOnly: true
          description: the user id which own this Scenario
        rootId:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          readOnly: true
          description: the scenario root id
        solutionId:
          type: string
          readOnly: true
          description: the Solution Id associated with this Scenario
        runTemplateId:
          type: string
          description: the Solution Run Template Id associated with this Scenario
        organizationId:
            type: string
            x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
            readOnly: true
            description: the associated Organization Id
        workspaceId:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          readOnly: true
          description: the associated Workspace Id
        state:
          $ref: '#/components/schemas/ScenarioJobState'
        creationDate:
          type: integer
          format: int64
          readOnly: true
          description: the Scenario creation date
        lastUpdate:
          type: integer
          format: int64
          readOnly: true
          description: the last time a Scenario was updated
        ownerName:
          type: string
          readOnly: true
          description: the name of the owner
        solutionName:
          type: string
          readOnly: true
          description: the Solution name
        runTemplateName:
          type: string
          readOnly: true
          description: the Solution Run Template name associated with this Scenario
        datasetList:
          type: array
          description: the list of Dataset Id associated to this Scenario Run Template
          items:
            type: string
        runSizing:
          description: definition of resources needed for the scenario run
          $ref: "#/components/schemas/ScenarioResourceSizing"
        parametersValues:
          type: array
          description: the list of Solution Run Template parameters values
          items:
            $ref: '#/components/schemas/ScenarioRunTemplateParameterValue'
        lastRun:
          allOf:
            - $ref: '#/components/schemas/ScenarioLastRun'
            - type: object
              description: the last Scenario Run for this Scenario
        parentLastRun:
          allOf:
            - $ref: '#/components/schemas/ScenarioLastRun'
            - type: object
              description: the last Scenario Run for the parent of this Scenario
        rootLastRun:
          allOf:
            - $ref: '#/components/schemas/ScenarioLastRun'
            - type: object
              description: the last Scenario Run for the root (master) of Scenario
        validationStatus:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Searchable"
          allOf:
            - $ref: '#/components/schemas/ScenarioValidationStatus'
        security:
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          allOf:
            - $ref: '#/components/schemas/ScenarioSecurity'
#      required:
#        - name
#        - runTemplateId
    ScenarioSecurity:
      type: object
      description: the Scenario security information
      properties:
        default:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the role by default
        accessControlList:
          type: array
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the list which can access this Scenario with detailed access control information
          items:
            $ref: '#/components/schemas/ScenarioAccessControl'
      required:
        - default
        - accessControlList
    ScenarioAccessControl:
      type: object
      description: a Scenario access control item
      properties:
        id:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: the identity id
        role:
          type: string
          x-field-extra-annotation: "@com.redis.om.spring.annotations.Indexed"
          description: a role
      required:
        - id
        - role
    ScenarioRole:
      type: object
      description: the Scenario Role
      properties:
        role:
          type: string
          description: the Scenario Role
      required:
        - role
    ScenarioResourceSizing:
      type: object
      description: a description object for resource requests and limits (default same configuration as basic sizing)
      properties:
        requests:
          $ref: '#/components/schemas/ResourceSizeInfo'
        limits:
          $ref: '#/components/schemas/ResourceSizeInfo'
      required:
        - requests
        - limits
    ResourceSizeInfo:
      type: object
      description: define cpus and memory needs
      properties:
        cpu:
          type: string
          description: define cpu needs
        memory:
          type: string
          description: define memory needs
      required:
        - cpu
        - memory
    ScenarioLastRun:
      type: object
      properties:
        scenarioRunId:
          type: string
          description: the last Scenario Run id
        csmSimulationRun:
          type: string
          description: the last Cosmo Tech Simulation Run id
        workflowId:
          type: string
          description: the last Workflow Id
        workflowName:
          type: string
          description: the last Workflow name
    ScenarioRunTemplateParameterValue:
      type: object
      description: the value of a Solution Run Template parameter for a Scenario
      properties:
        parameterId:
          type: string
          description: the parameter Id
        varType:
          type: string
          readOnly: true
          description: the parameter value type
        value:
          type: string
          description: the parameter value
        isInherited:
          type: boolean
          description: whether or not the value is inherited from parent or has been changed
      required:
        - parameterId
        - value
    ScenarioComparisonResult:
      type: object
      description: the result of the comparison of two scenarios
      properties:
        scenarioId:
          type: string
          readOnly: true
          description: the Scenario Id which is the reference for the comparison
        comparedScenarioId:
          type: string
          readOnly: true
          description: the Scenario Id the reference Scenario is compared to
        changedValues:
          type: array
          readOnly: true
          description: the list of changed values for parameters
          items:
            $ref: '#/components/schemas/ScenarioChangedParameterValue'
    ScenarioChangedParameterValue:
      type: object
      description: the difference between the values of a parameter
      properties:
        parameterId:
          type: string
          readOnly: true
          description: the parameter id the values refer to
        varType:
          type: string
          readOnly: true
          description: the parameter value type
        value:
          type: string
          readOnly: true
          description: the parameter value for the reference Scenario
        comparedValue:
          type: string
          readOnly: true
          description: the parameter value for the compared Scenario
    ScenarioDataDownloadInfo:
      type: object
      description: Scenario data download job info
      properties:
        url:
          type: string
          readOnly: true
          description: the Scenario Data Download URL
        state:
          $ref: '#/components/schemas/ScenarioJobState'
    ScenarioDataDownloadJob:
      type: object
      description: Scenario data download job
      properties:
        id:
          type: string
          readOnly: true
          description: the Scenario Data Download job identifier
    ScenarioJobState:
      type: string
      readOnly: true
      description: the Scenario job state
      enum:
        - Created
        - Running
        - Successful
        - Failed
        # PROD-7888 : When requesting the scenario state right after a run has been submitted,
        # the scenario run service (e.g., Argo Workflow) might not have scheduled the run
        # effectively yet.
        # Furthermore, temporary communication errors might occur anytime when remotely
        # fetching last scenario run statuses.
        - Unknown
        # PROD-7420 : return data ingestion status
        - DataIngestionInProgress
    ScenarioValidationStatus:
      type: string
      description: the validation status of the scenario
      enum:
        - Draft
        - Rejected

As a result of creating your Scenario you will get an identifier of the form s-xxxxxxxx that you can use to run it (as we will see further) and reference it in other endpoints.

A scenario essentially combines a run template with a dataset. The run template provides a set of parameters that can then be modified by the user before running the scenario. Multiple elements are required for defining a scenario, most notably:

  • runTemplateId: should be one of the run template identifiers defined in our Solution.
  • datasetList: should reference a previously defined Dataset using its identifier.
  • parametersValues: is a list of objects representing the parameters we defined in the Solution, with the actual values they are set to.

Run our distant Scenario

We will need to either use a lot of parameters or environment variables. To make things easier we can make use of a parameter of csm-orc run to generate a template of .env file we will be able to use.

That parameter is --gen-env-target and takes a filename to create a .env file initialized with all the environment variables defined in our run.json

If a variable is set multiple times, only the last one will be taken into account. Each environment variable will have one and only one value from the following list (highest value in the list takes precedence):

  • The value defined in the run.json.
  • The current value in your working environment.
  • The defaultValue defined in the run.json.
  • The description defined in the run.json.
  • None if no other value can be found.
use of --gen-env-target
csm-orc run code/run_templates/orchestrator_tutorial_2/run.json --gen-env-target code/run_templates/orchestrator_tutorial_2/vars.env
cat code/run_templates/orchestrator_tutorial_2/vars.env
# CSM_API_SCOPE=The scope of identification used to request access token for your Cosmo Tech API instance
# CSM_API_URL=The URL used to query your Cosmo Tech API instance
# CSM_DATASET_ABSOLUTE_PATH=Simulation/Resource/scenariorun-data
# CSM_ORGANIZATION_ID=The identifier of the organization in the Cosmo Tech API
# CSM_PARAMETERS_ABSOLUTE_PATH=code/run_templates/orchestrator_tutorial_2
# CSM_SCENARIO_ID=The identifier of the scenario in the Cosmo Tech API
# CSM_SIMULATION=BusinessApp_Simulation
# CSM_WORKSPACE_ID=The id of the workspace in the Cosmo Tech API

This .env file can then be used as a parameter of the docker run command with --env-file or can be used with a tool like dotenv (pip install dotenv) to temporary set the environment variables of commands. You can also just load it to get all values in your current environment.

set -o allexport
source code/run_templates/orchestrator_tutorial_2/vars.env set
+o allexport
csm-orc run code/run_templates/orchestrator_tutorial_2/run.json

docker run --env-file code/run_templates/orchestrator_tutorial_2/vars.env cosmotech/mybrewery_simulator

dotenv -e code/run_templates/orchestrator_tutorial_2/vars.env csm-orc run code/run_templates/orchestrator_tutorial_2/run.json

I want to use my local data instead of needing a Scenario

The csm-orc run command comes with an optional parameter --skip-step that can be set in an environment variable CSM_SKIP_STEPS. This parameter can be given a list of steps that will be ignored during the run.

Since the "only" step used to download distant data from our run is DownloadScenarioData we can simply run locally by skipping it:

Running only the SimulationRun step
csm-orc run --skip-step DownloadScenarioData --skip-step ApplyParameters code/run_templates/orchestrator_tutorial_2/run.json