Concerning configuration¶

Objective

Add environment variables to our script
Use environment variables in our orchestration
Use CommandTemplate to combine close commands

Taking a look at Environment Variables¶

We will start in the same folder as the previous tutorial : MyFirstOrchestrationProject and augment it to use some Environment Variables to configure our commands.

First we will start by modifying our scripts to now accept an environment variable for the file path

FibonacciDisplay

fibonacci.py
import argparse
import os

parser = argparse.ArgumentParser(description="Fibonacci printer")
parser.add_argument("n",
                    type=int,
                    help="The max rank of the fibonacci sequence to write")
parser.add_argument("--filename",
                    type=argparse.FileType('w'),
                    help="The file to write to",
                    default=os.environ.get("FIBO_FILE_PATH"))


def fibonacci_sequence(n: int = 0):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b


if __name__ == "__main__":
    args = parser.parse_args()
    with args.filename as f:
        for v in fibonacci_sequence(args.n):
            f.write(f"{v}\n")

display_file.py
import argparse
import os

parser = argparse.ArgumentParser(description="File printer")
parser.add_argument("--filename",
                    type=argparse.FileType('r'),
                    help="The file to print",
                    default=os.environ.get("FIBO_FILE_PATH"))

if __name__ == "__main__":
    args = parser.parse_args()

    with args.filename as f:
        for v in f.readlines():
            print(v.strip())

Those modifications will allow us to set an Environment Variable FIBO_FILE_PATH that will be used for our file name

Let's try our scripts

Running those two files is easy

export FIBO_FILE_PATH=fib_second.txt
python fibonacci.py 10
python display_file.py

Without environment variable we would run the following commands

python fibonacci.py 10 --filename fib_second.txt
python display_file.py --filename fib_second.txt

Both ways should display the following lines

Now that our commands work we will look at the orchestration file to configure those environment variables

run_env.json

{
  "steps": [
    {
      "id": "run-fibo",
      "command": "python",
      "arguments": [ "fibonacci.py", "10" ],
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file run-fibo will write to"
        }
      }
    },
    {
      "id": "run-display",
      "command": "python",
      "arguments": [ "display_file.py" ],
      "precedents": [ "run-fibo" ],
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file run-display will read and print to stdout"}
      }
    }
  ]
}

We added 2 definitions of our FIBO_FILE_PATH to the steps, so we can try to run our script

Run the orchestrator without the environment variable

# First we remove the definition of FIBO_FILE_PATH from the environment for the example
unset FIBO_FILE_PATH
csm-orc run run_env.json
# [YYYY/MM/DD-HH:mm:SS] ERROR    Missing environment values
# [YYYY/MM/DD-HH:mm:SS] ERROR     - FIBO_FILE_PATH 
# [YYYY/MM/DD-HH:mm:SS] ERROR    Missing environment variables, check the logs

We can see that without defining our environment variable issues are displayed before the run.

If we wanted to know which environment variables are required for our orchestration script we can do the following

Getting information about environment variables

csm-orc run run_env.json --display-env
# [YYYY/MM/DD-HH:mm:SS] INFO     Environment variable defined for run_env.json
# [YYYY/MM/DD-HH:mm:SS] INFO      - FIBO_FILE_PATH:
#                                   - A file run-fibo will write to
#                                   - A file run-display will read and print to stdout

We can see that all descriptions of a variable are made available.

Let's give a value to FIBO_FILE_PATH and run our command

Run the orchestrator with the environment variable

FIBO_FILE_PATH=fib_second.txt csm-orc run run_env.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# 21
# 34
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     run-fibo (Done)
# [YYYY/MM/DD-HH:mm:SS] INFO     run-display (Done)

We can see that our orchestrator works now.

Use Environment Variables as Argument¶

To add more configuration to our file lets use an environment variable for the run-fibo step argument.

run_env_arg.json

{
  "steps": [
    {
      "id": "run-fibo",
      "command": "python",
      "arguments": [ "fibonacci.py", "$FIBO_COUNT" ],
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file run-fibo will write to"
        },
        "FIBO_COUNT": {
          "description": "The rank of the fibonacci sequence run-fibo will write to",
          "defaultValue": "10"
        }
      }
    },
    {
      "id": "run-display",
      "command": "python",
      "arguments": [ "display_file.py" ],
      "precedents": [ "run-fibo" ],
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file run-display will read and print to stdout"}
      }
    }
  ]
}

In this file we defined an environment variable that will be used as an argument for our command (by using it as an argument preceded by $), that way we don't need to modify our script.

We also defined a defaultValue for the argument, ensuring that even if the environment variable is not defined a default value is used.

Run without the new variableRun with the new variable

Run as previously

export FIBO_FILE_PATH=fib_second.txt
csm-orc run run_env_arg.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# 21
# 34
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     run-fibo (Done)
# [YYYY/MM/DD-HH:mm:SS] INFO     run-display (Done)

Run the orchestrator with the new environment variable

export FIBO_FILE_PATH=fib_second.txt
export FIBO_COUNT=8
csm-orc run run_env_arg.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     run-fibo (Done)
# [YYYY/MM/DD-HH:mm:SS] INFO     run-display (Done)

Use CommandTemplate to reduce copy¶

We now have 2 steps that use the same base command and a common Environment Variable. Let's make use of the CommandTemplate to reduce the number of time we need to impact our steps.

run_with_template.json

{
  "steps": [
    {
      "id": "run-fibo",
      "commandId": "python-with-fibo-file",
      "arguments": [ "fibonacci.py", "$FIBO_COUNT" ],
      "environment": {
        "FIBO_COUNT": {
          "description": "The rank of the fibonacci sequence run-fibo will write to",
          "defaultValue": "10"
        }
      }
    },
    {
      "id": "run-display",
      "commandId": "python-with-fibo-file",
      "arguments": [ "display_file.py" ],
      "precedents": [ "run-fibo" ]
    }
  ],
  "commandTemplates": [
    {
      "id": "python-with-fibo-file",
      "command": "python",
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file available to the command"
        }
      }
    }
  ]
}

We grouped the common part of the steps in a new command template called python-with-fibo-file, then replaced the command of our steps by its commandId.

Now we can call the new file as previously

Run the orchestrator with a command template

export FIBO_FILE_PATH=fib_second.txt
export FIBO_COUNT=8 
csm-orc run run_with_template.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     run-fibo (Done)
# [YYYY/MM/DD-HH:mm:SS] INFO     run-display (Done)

Now you can create command templates, use environment variables to configure your scripts, and set some values for those.

Make use of optional Environment Variables¶

In some case you may want to have optional effects that can be used to change your script only if present (the use of a defaultValue does not make sense for those)

For example, we will make a change to the run-display step to either print the content of the file as is, or if an Environment Variable DISPLAY_SYMBOL is set it will instead display that symbol on each line X times, where X is the number read from the file.

So DISPLAY_SYMBOL=X and FIBO_COUNT=5 would look like that:

DISPLAY_SYMBOL=X and FIBO_COUNT=5

#
# X
# X
# XX
# XXX

updated_display.py
import argparse
import os

parser = argparse.ArgumentParser(description="File printer")
parser.add_argument("--filename",
                    type=argparse.FileType('r'),
                    help="The file to print",
                    default=os.environ.get("FIBO_FILE_PATH"))

display_symbol = os.environ.get("DISPLAY_SYMBOL")

if __name__ == "__main__":
    args = parser.parse_args()

    with args.filename as f:
        for v in f.readlines():
            value = int(v.strip())
            if display_symbol:
                print(display_symbol * value)
            else:
                print(value)

And to make use of that new script we can update our .json file as following

optional_envvar.json

{
  "steps": [
    {
      "id": "run-fibo",
      "commandId": "python-with-fibo-file",
      "arguments": [ "fibonacci.py", "$FIBO_COUNT" ],
      "environment": {
        "FIBO_COUNT": {
          "description": "The rank of the fibonacci sequence run-fibo will write to",
          "defaultValue": "10"
        }
      }
    },
    {
      "id": "run-display",
      "commandId": "python-with-fibo-file",
      "arguments": [ "updated_display.py" ],
      "environment": {
        "DISPLAY_SYMBOL": {
          "optional": true,
          "description": "A symbol used to replace the display of a number by a repetition of it."
        }
      },
      "precedents": [ "run-fibo" ]
    }
  ],
  "commandTemplates": [
    {
      "id": "python-with-fibo-file",
      "command": "python",
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file available to the command"
        }
      }
    }
  ]
}

Run without DISPLAY_SYMBOLRun with DISPLAY_SYMBOL

export FIBO_FILE_PATH=fib_second.txt
export FIBO_COUNT=8 
csm-orc run optional_envvar.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     run-fibo (Done)
# [YYYY/MM/DD-HH:mm:SS] INFO     run-display (Done)

export FIBO_FILE_PATH=fib_second.txt
export FIBO_COUNT=8 
export DISPLAY_SYMBOL=X 
csm-orc run optional_envvar.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 
# X
# X
# XX
# XXX
# XXXXX
# XXXXXXXX
# XXXXXXXXXXXXX
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     run-fibo (Done)
# [YYYY/MM/DD-HH:mm:SS] INFO     run-display (Done)