Skip to content

My first orchestration

Objective

  • Set up an orchestration project
  • Create a few scripts to organize
  • Create an orchestration file to run our scripts

Setting up our project

During this tutorial we will start from a clean installation of csm-orc and work our way to an orchestrated solution.

Set up our project
# First we will create a new folder for our project
mkdir MyFirstOrchestrationProject
cd MyFirstOrchestrationProject
# Now that we are in our project folder we will set up the orchestrator using a python venv
# We create the venv in the folder `.venv`
python -m venv .venv
# We activate the venv
. .venv/bin/activate
# Now we can install the orchestrator using pip
pip install cosmotech-run-orchestrator
# We can check that our installation worked by running the orchestrator help
csm-orc --help

After all that our project is ready to start

Creating our first scripts

In this part we will create 2 simple python script that will interact by using a common file, the first script should write the first N members of the Fibonacci sequence on the file, and the second one will display them.

fibonacci.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
import argparse

parser = argparse.ArgumentParser(description="Fibonacci printer")
parser.add_argument("n", type=int, help="The max rank of the fibonacci sequence to write")
parser.add_argument("filename", type=argparse.FileType("w"), help="The file to write to")


def fibonacci_sequence(n: int = 0):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b


if __name__ == "__main__":
    args = parser.parse_args()
    with args.filename as f:
        for v in fibonacci_sequence(args.n):
            f.write(f"{v}\n")
display_file.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
import argparse

parser = argparse.ArgumentParser(description="File printer")
parser.add_argument("filename", type=argparse.FileType("r"), help="The file to print")

if __name__ == "__main__":
    args = parser.parse_args()

    with args.filename as f:
        for v in f.readlines():
            print(v.strip())

Let's try our scripts

Running those two files is easy 

python fibonacci.py 10 fib.txt
python display_file.py fib.txt
This should display the following lines 
0
1
1
2
3
5
8
13
21
34

Write a first orchestration

Now that we have our two files, we want to create an orchestration file to run them.

An orchestration file is a json file following the given JSON schema

JSON-schema

The following schema can be impressive, but we will go through most of it in the next points.

JSON-schema
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://cosmotech.com/run_template.schema.json",
  "title": "Run Template",
  "description": "A run template description",
  "type": "object",
  "properties": {
    "commandTemplates": {
      "description": "A list of Commands Templates",
      "type": "array",
      "items": {
        "description": "A Command Template describe a single executable with default properties",
        "type": "object",
        "properties": {
          "id": {
            "type": "string",
            "description": "The Id of the Command Template",
            "pattern": "^\\S*$"
          },
          "description": {
            "type": "string",
            "description": "A description of the command template"
          },
          "command": {
            "type": "string",
            "description": "The root bash command necessary to execute the template"
          },
          "arguments": {
            "type": "array",
            "description": "The list of default arguments passed to the command",
            "items": {
              "type": "string"
            }
          },
          "useSystemEnvironment": {
            "type": "boolean",
            "description": "Should the system environment be fully passed to the command ?"
          },
          "environment": {
            "type": "object",
            "description": "The default list of Environment Variables required for the command",
            "patternProperties": {
              ".+": {
                "$ref": "#/$defs/environmentVariable"
              }
            },
            "minProperties": 1
          }
        },
        "additionalProperties": false,
        "required": [
          "id",
          "command"
        ]
      }
    },
    "steps": {
      "description": "A list of Steps descriptors",
      "type": "array",
      "items": {
        "description": "A Step is a single instance of a Command that is scheduled to be run, can use an existing command or define its own",
        "type": "object",
        "properties": {
          "id": {
            "type": "string",
            "description": "The Id of the Step",
            "pattern": "^\\S*$"
          },
          "description": {
            "type": "string",
            "description": "A description of the step (override command template description)"
          },
          "commandId": {
            "type": "string",
            "description": "An Id for an existing command"
          },
          "command": {
            "type": "string",
            "description": "The root bash command necessary to execute the command"
          },
          "arguments": {
            "type": "array",
            "description": "The list of arguments passed to the command (replace the default one)",
            "items": {
              "type": "string"
            }
          },
          "useSystemEnvironment": {
            "type": "boolean",
            "description": "Should the system environment be fully passed to the command ?"
          },
          "environment": {
            "type": "object",
            "description": "The list of Environment Variables defined for the command (replace the default one)",
            "patternProperties": {
              ".+": {
                "$ref": "#/$defs/environmentVariable"
              }
            },
            "minProperties": 1
          },
          "precedents": {
            "type": "array",
            "description": "A list of steps that have to be run before this one",
            "items": {
              "type": "string"
            }
          },
          "outputs": {
            "type": "object",
            "description": "Defines named outputs that this step produces",
            "patternProperties": {
              ".+": {
                "type": "object",
                "properties": {
                  "description": {
                    "type": "string",
                    "description": "Description of what this output represents"
                  },
                  "defaultValue": {
                    "type": "string",
                    "description": "Default value if no output is produced"
                  },
                  "optional": {
                    "type": "boolean",
                    "description": "Whether this output is optional",
                    "default": false
                  },
                  "hidden": {
                    "type": "boolean",
                    "description": "Whether this output should be hidden from logs",
                    "default": false
                  }
                }
              }
            }
          },
          "inputs": {
            "type": "object",
            "description": "Maps input names to outputs from other steps",
            "patternProperties": {
              ".+": {
                "type": "object",
                "properties": {
                  "stepId": {
                    "type": "string",
                    "description": "ID of the step to take output from"
                  },
                  "output": {
                    "type": "string",
                    "description": "Name of the output to use from the referenced step"
                  },
                  "as": {
                    "type": "string",
                    "description": "Name of environment variable to receive the input"
                  },
                  "defaultValue": {
                    "type": "string",
                    "description": "Default value if the referenced output is not available"
                  },
                  "optional": {
                    "type": "boolean",
                    "description": "Whether this input is optional",
                    "default": false
                  },
                  "hidden": {
                    "type": "boolean",
                    "description": "Whether this input should be hidden from logs",
                    "default": false
                  }
                },
                "required": ["stepId", "output", "as"]
              }
            }
          }
        },
        "additionalProperties": false,
        "oneOf": [
          {
            "required": [
              "id",
              "command"
            ]
          },
          {
            "required": [
              "id",
              "commandId"
            ]
          }
        ]
      }
    }
  },
  "required": [
    "steps"
  ],
  "$defs": {
    "environmentVariable": {
      "type": "object",
      "description": "A environment variable descriptor",
      "properties": {
        "defaultValue": {
          "type": "string",
          "description": "The default value of the required variable, if not set, the variable has to be set in the system"
        },
        "value": {
          "type": "string",
          "description": "The effective value of the required variable, will override any system value"
        },
        "description": {
          "type": "string",
          "description": "A description of the required Environment Variable for documentation reasons"
        },
        "optional": {
          "type": "boolean",
          "description": "Should the Environment Variable be required for the run of a step ?"
        }
      },
      "additionalProperties": false
    }
  }
}

The first step

In the schema we can see that it is divided in two parts :

  • the CommandTemplate
  • the Step

For this first orchestration file we will only use steps.

Let's take a look at an example step

simple_step.json
1
2
3
4
5
6
7
8
9
{
  "steps": [
    {
      "id": "echo-foo-bar",
      "command": "echo",
      "arguments": [ "foo", "bar" ]
    }
  ]
}

In this minimal file we can see one step with the id echo-foo-bar that runs the command echo with the arguments foo and bar.

This orchestration file is valid following the JSON schema as we have at least 1 element in steps, and the element as at least an id and a command

We can run this orchestration file with the following command :

run simple_step.json
1
2
3
4
5
6
7
csm-orc run simple_step.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step echo-foo-bar
# foo bar
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step echo-foo-bar
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     echo-foo-bar (Done)

On line 4 we can see the result of the command echo foo bar which is defined in the step.

Now using this basis we will write our commands in an orchestration file using the same options as the previous example

Let's try our scripts as an orchestration file

run.json
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
{
  "steps": [
    {
      "id": "run-fibo",
      "command": "python",
      "arguments": [ "fibonacci.py", "10", "fib.txt" ]
    },
    {
      "id": "run-display",
      "command": "python",
      "arguments": [ "display_file.py", "fib.txt" ],
      "precedents": [ "run-fibo" ]
    }
  ]
}
Now we can run our file using the orchestrator
run our first orchestration
csm-orc run run.json

In the example run.json you can see on line 12 the apparition of the key-words precedents, it is used to order our operations.

Here by setting the step run-fibo as a precedent to the step run-display we ensure that the first script will run before the second.

And now we created a simple example of orchestration file to run some of our scripts. In the next tutorial we will look at how to use CommandTemplates to re-use possibly complex commands, and Environment Variables to change the effect of our commands.