Skip to content

Concerning configuration

Objective

  • Add environment variables to our script
  • Use environment variables in our orchestration
  • Use CommandTemplate to combine close commands

Taking a look at Environment Variables

We will start in the same folder as the previous tutorial : MyFirstOrchestrationProject and augment it to use some Environment Variables to configure our commands.

First we will start by modifying our scripts to now accept an environment variable for the file path

fibonacci.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import argparse
import os

parser = argparse.ArgumentParser(description="Fibonacci printer")
parser.add_argument("n",
                    type=int,
                    help="The max rank of the fibonacci sequence to write")
parser.add_argument("--filename",
                    type=argparse.FileType('w'),
                    help="The file to write to",
                    default=os.environ.get("FIBO_FILE_PATH"))


def fibonacci_sequence(n: int = 0):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b


if __name__ == "__main__":
    args = parser.parse_args()
    with args.filename as f:
        for v in fibonacci_sequence(args.n):
            f.write(f"{v}\n")
display_file.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import argparse
import os

parser = argparse.ArgumentParser(description="File printer")
parser.add_argument("--filename",
                    type=argparse.FileType('r'),
                    help="The file to print",
                    default=os.environ.get("FIBO_FILE_PATH"))

if __name__ == "__main__":
    args = parser.parse_args()

    with args.filename as f:
        for v in f.readlines():
            print(v.strip())

Those modifications will allow us to set an Environment Variable FIBO_FILE_PATH that will be used for our file name

Let's try our scripts

Running those two files is easy 

export FIBO_FILE_PATH=fib_second.txt
python fibonacci.py 10
python display_file.py
Without environment variable we would run the following commands 
python fibonacci.py 10 --filename fib_second.txt
python display_file.py --filename fib_second.txt
Both ways should display the following lines 
0
1
1
2
3
5
8
13
21
34

Now that our commands work we will look at the orchestration file to configure those environment variables

run_env.json
{
  "steps": [
    {
      "id": "run-fibo",
      "command": "python",
      "arguments": [ "fibonacci.py", "10" ],
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file run-fibo will write to"
        }
      }
    },
    {
      "id": "run-display",
      "command": "python",
      "arguments": [ "display_file.py" ],
      "precedents": [ "run-fibo" ],
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file run-display will read and print to stdout"}
      }
    }
  ]
}

We added 2 definitions of our FIBO_FILE_PATH to the steps, so we can try to run our script

Run the orchestrator without the environment variable
# First we remove the definition of FIBO_FILE_PATH from the environment for the example
unset FIBO_FILE_PATH
csm-orc run run_env.json
# [YYYY/MM/DD-HH:mm:SS] ERROR    Missing environment values
# [YYYY/MM/DD-HH:mm:SS] ERROR     - FIBO_FILE_PATH 
# [YYYY/MM/DD-HH:mm:SS] ERROR    Missing environment variables, check the logs

We can see that without defining our environment variable issues are displayed before the run.

If we wanted to know which environment variables are required for our orchestration script we can do the following

Getting information about environment variables
csm-orc run run_env.json --display-env
# [YYYY/MM/DD-HH:mm:SS] INFO     Environment variable defined for run_env.json
# [YYYY/MM/DD-HH:mm:SS] INFO      - FIBO_FILE_PATH:
#                                   - A file run-fibo will write to
#                                   - A file run-display will read and print to stdout

We can see that all descriptions of a variable are made available.

Let's give a value to FIBO_FILE_PATH and run our command

Run the orchestrator with the environment variable
FIBO_FILE_PATH=fib_second.txt csm-orc run run_env.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# 21
# 34
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Step run-fibo
#                                Command: python fibonacci.py 10
#                                Environment:
#                                - FIBO_FILE_PATH: A file run-fibo will write to
#                                Status: Done
# [YYYY/MM/DD-HH:mm:SS] INFO     Step run-display
#                                Command: python display_file.py
#                                Environment:
#                                - FIBO_FILE_PATH: A file run-display will read and print to stdout

We can see that our orchestrator works now.

Use Environment Variables as Argument

To add more configuration to our file lets use an environment variable for the run-fibo step argument.

run_env_arg.json
{
  "steps": [
    {
      "id": "run-fibo",
      "command": "python",
      "arguments": [ "fibonacci.py", "$FIBO_COUNT" ],
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file run-fibo will write to"
        },
        "FIBO_COUNT": {
          "description": "The rank of the fibonacci sequence run-fibo will write to",
          "defaultValue": "10"
        }
      }
    },
    {
      "id": "run-display",
      "command": "python",
      "arguments": [ "display_file.py" ],
      "precedents": [ "run-fibo" ],
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file run-display will read and print to stdout"}
      }
    }
  ]
}

In this file we defined an environment variable that will be used as an argument for our command (by using it as an argument preceded by $), that way we don't need to modify our script.

We also defined a defaultValue for the argument, ensuring that even if the environment variable is not defined a default value is used.

Run as previously
export FIBO_FILE_PATH=fib_second.txt
csm-orc run run_env_arg.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# 21
# 34
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Step run-fibo
#                                Command: python fibonacci.py 10
#                                Environment:
#                                - FIBO_FILE_PATH: A file run-fibo will write to
#                                - FIBO_COUNT: The rank of the fibonacci sequence run-fibo will write to
#                                Status: Done
# [YYYY/MM/DD-HH:mm:SS] INFO     Step run-display
#                                Command: python display_file.py
#                                Environment:
#                                - FIBO_FILE_PATH: A file run-display will read and print to stdout
Run the orchestrator with the new environment variable
export FIBO_FILE_PATH=fib_second.txt
export FIBO_COUNT=8
csm-orc run run_env_arg.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Step run-fibo
#                                Command: python fibonacci.py 10
#                                Environment:
#                                - FIBO_FILE_PATH: A file run-fibo will write to
#                                - FIBO_COUNT: The rank of the fibonacci sequence run-fibo will write to
#                                Status: Done
# [YYYY/MM/DD-HH:mm:SS] INFO     Step run-display
#                                Command: python display_file.py
#                                Environment:
#                                - FIBO_FILE_PATH: A file run-display will read and print to stdout

Use CommandTemplate to reduce copy

We now have 2 steps that use the same base command and a common Environment Variable. Let's make use of the CommandTemplate to reduce the number of time we need to impact our steps.

run_with_template.json
{
  "steps": [
    {
      "id": "run-fibo",
      "commandId": "python-with-fibo-file",
      "arguments": [ "fibonacci.py", "$FIBO_COUNT" ],
      "environment": {
        "FIBO_COUNT": {
          "description": "The rank of the fibonacci sequence run-fibo will write to",
          "defaultValue": "10"
        }
      }
    },
    {
      "id": "run-display",
      "commandId": "python-with-fibo-file",
      "arguments": [ "display_file.py" ],
      "precedents": [ "run-fibo" ]
    }
  ],
  "commandTemplates": [
    {
      "id": "python-with-fibo-file",
      "command": "python",
      "environment": {
        "FIBO_FILE_PATH": {
          "description": "A file available to the command"
        }
      }
    }
  ]
}

We grouped the common part of the steps in a new command template called python-with-fibo-file, then replaced the command of our steps by its commandId.

Now we can call the new file as previously

Run the orchestrator with a command template
export FIBO_FILE_PATH=fib_second.txt
export FIBO_COUNT=8 
csm-orc run run_with_template.json
# [YYYY/MM/DD-HH:mm:SS] INFO     ===      Run     ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-fibo
# [YYYY/MM/DD-HH:mm:SS] INFO     Starting step run-display
# 0
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# [YYYY/MM/DD-HH:mm:SS] INFO     Done running step run-display
# [YYYY/MM/DD-HH:mm:SS] INFO     ===     Results    ===
# [YYYY/MM/DD-HH:mm:SS] INFO     Step run-fibo
#                                Command: python fibonacci.py $FIBO_COUNT
#                                Environment:
#                                - FIBO_COUNT: The rank of the fibonacci sequence run-fibo will write to
#                                - FIBO_FILE_PATH: A file available to the command
#                                Status: Done
# [YYYY/MM/DD-HH:mm:SS] INFO     Step run-display
#                                Command: python display_file.py
#                                Environment:
#                                - FIBO_FILE_PATH: A file available to the command
#                                Status: Done

Now you can create command templates, use environment variables to configure your scripts, and set some values for those.