csm-data¶
Cosmo Tect Data Interface
Command toolkit provinding quick implementation of data connections to use inside the Cosmo Tech Platform
Usage:
csm-data [OPTIONS] COMMAND [ARGS]...
Options:
--log-level LVL Either CRITICAL, ERROR, WARNING, INFO or DEBUG [env var:
LOG_LEVEL]
--version Print version number and return.
--help Show this message and exit.
Subcommands
- api: Cosmo Tech API helper command
- s3-bucket-load: Download S3 bucket content to a given folder
csm-data api¶
Cosmo Tech API helper command
This command will inform you of which connection is available to use for the Cosmo Tech API
If no connection is available, will list all possible set of parameters and return an error code,
You can use this command in a csm-orc template to make sure that API connection is available.
Usage:
csm-data api [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Subcommands
- rds-load-csv: Download a CSV file from the Cosmo Tech Run API using a given SQL query
- rds-send-csv: Send all csv files from a folder to the results service of the Cosmo Tech API
- tdl-load-files: Query a twingraph and loads all the data from it
- tdl-send-files: Reads a folder CSVs and send those to the Cosmo Tech API as a Dataset
- wsf-load-file: Load files from a workspace inside the API
- wsf-send-file: Send a file to a workspace inside the API
csm-data api rds-load-csv¶
Download a CSV file from the Cosmo Tech Run API using a given SQL query
Requires a valid connection to the API to send the data
Usage:
csm-data api rds-load-csv [OPTIONS]
Options:
--target-folder PATH The folder where the csv will be written [env
var: CSM_DATASET_ABSOLUTE_PATH; required]
--organization-id o-XXXXXXXX An organization id for the Cosmo Tech API
[env var: CSM_ORGANIZATION_ID; required]
--workspace-id w-XXXXXXXX A workspace id for the Cosmo Tech API [env
var: CSM_WORKSPACE_ID; required]
--runner-id r-XXXXXXXX A runner id for the Cosmo Tech API [env var:
CSM_RUNNER_ID; required]
--run-id run-XXXXXX A run id for the Cosmo Tech API [env var:
CSM_RUN_ID; required]
--file-name NAME A file name to write the query results
[default: results; required]
--query SQL_QUERY A run id for the Cosmo Tech API [default:
SELECT table_name FROM
information_schema.tables WHERE
table_schema='public']
--help Show this message and exit.
csm-data api rds-send-csv¶
Send all csv files from a folder to the results service of the Cosmo Tech API
Requires a valid connection to the API to send the data
Usage:
csm-data api rds-send-csv [OPTIONS]
Options:
--source-folder PATH The folder containing csvs to send [env var:
CSM_DATASET_ABSOLUTE_PATH; required]
--organization-id o-XXXXXXXX An organization id for the Cosmo Tech API
[env var: CSM_ORGANIZATION_ID; required]
--workspace-id w-XXXXXXXX A workspace id for the Cosmo Tech API [env
var: CSM_WORKSPACE_ID; required]
--runner-id r-XXXXXXXX A runner id for the Cosmo Tech API [env var:
CSM_RUNNER_ID; required]
--run-id run-XXXXXX A run id for the Cosmo Tech API [env var:
CSM_RUN_ID; required]
--help Show this message and exit.
csm-data api tdl-load-files¶
Query a twingraph and loads all the data from it
Will create 1 csv file per node type / relationship type
The twingraph must have been populated using the "tdl-send-files" command for this to work correctly
Requires a valid connection to the API to send the data
Usage:
csm-data api tdl-load-files [OPTIONS]
Options:
--organization-id o-XXXXXXXX An organization id for the Cosmo Tech API
[env var: CSM_ORGANIZATION_ID; required]
--workspace-id w-XXXXXXXX A workspace id for the Cosmo Tech API [env
var: CSM_WORKSPACE_ID; required]
--scenario-id s-XXXXXXXX A scenario id for the Cosmo Tech API [env
var: CSM_SCENARIO_ID]
--runner-id r-XXXXXXXX A runner id for the Cosmo Tech API [env var:
CSM_RUNNER_ID]
--dir PATH Path to the directory to write the results to
[env var: CSM_DATASET_ABSOLUTE_PATH; required]
--help Show this message and exit.
csm-data api tdl-send-files¶
Reads a folder CSVs and send those to the Cosmo Tech API as a Dataset
CSVs must follow a given format :
- Nodes files must have an id column
- Relationship files must have id, src and dest columns
Non-existing relationship (aka dest or src does not point to existing node) won't trigger an error, the relationship will not be created instead.
Requires a valid connection to the API to send the data
Usage:
csm-data api tdl-send-files [OPTIONS]
Options:
--api-url URI The URI to a Cosmo Tech API instance [env
var: CSM_API_URL; required]
--organization-id o-XXXXXXXX An organization id for the Cosmo Tech API
[env var: CSM_ORGANIZATION_ID; required]
--workspace-id w-XXXXXXXX A workspace id for the Cosmo Tech API [env
var: CSM_WORKSPACE_ID; required]
--runner-id r-XXXXXXXX A runner id for the Cosmo Tech API [env var:
CSM_RUNNER_ID; required]
--dir PATH Path to the directory containing csvs to send
[env var: CSM_DATASET_ABSOLUTE_PATH; required]
--clear / --keep Flag to clear the target dataset first (if set
to True will clear the dataset before sending
anything, irreversibly) [default: clear]
--help Show this message and exit.
csm-data api wsf-load-file¶
Load files from a workspace inside the API
Requires a valid connection to the API to send the data
Usage:
csm-data api wsf-load-file [OPTIONS]
Options:
--organization-id o-XXXXXXXX An organization id for the Cosmo Tech API
[env var: CSM_ORGANIZATION_ID; required]
--workspace-id w-XXXXXXXX A workspace id for the Cosmo Tech API [env
var: CSM_WORKSPACE_ID; required]
--workspace-path PATH Path inside the workspace to load (end with
'/' for a folder)
--target-folder PATH Folder in which to send the downloaded file
[env var: CSM_DATASET_ABSOLUTE_PATH; required]
--help Show this message and exit.
csm-data api wsf-send-file¶
Send a file to a workspace inside the API
Requires a valid connection to the API to send the data
This implementation make use of an API Key
Usage:
csm-data api wsf-send-file [OPTIONS]
Options:
--organization-id o-XXXXXXXX An organization id for the Cosmo Tech API
[env var: CSM_ORGANIZATION_ID; required]
--workspace-id w-XXXXXXXX A workspace id for the Cosmo Tech API [env
var: CSM_WORKSPACE_ID; required]
--file-path PATH Path to the file to send as a workspace file
[required]
--workspace-path PATH Path inside the workspace to store the file
(end with '/' for a folder) [required]
--overwrite / --keep Flag to overwrite the target file [default:
overwrite]
--help Show this message and exit.
csm-data s3-bucket-load¶
Download S3 bucket content to a given folder
Make use of the default AWS/S3 configuration to access the bucket
More information is available on this page: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
The following environment variables can be used to configure the connection:
- AWS_ENDPOINT_URL : The uri pointing to the S3 service endpoint
- AWS_ACCESS_KEY_ID : Your access key to the service
- AWS_SECRET_ACCESS_KEY : The secret associated to the access key
Usage:
csm-data s3-bucket-load [OPTIONS]
Options:
--target-folder PATH The folder in which to download the bucket content
[env var: CSM_DATASET_ABSOLUTE_PATH; required]
--bucket-name BUCKET The bucket on S3 to download [env var:
CSM_DATA_BUCKET_NAME; required]
--help Show this message and exit.