load-csv-folder¶
Help command
> csm-data custom load-csv-folder --help
Usage: csm-data custom load-csv-folder [OPTIONS]
Stream every CSV file of a folder into its own existing PostgreSQL table with
COPY FROM STDIN
Each .csv file directly in the folder (not recursive) is loaded into the table
named table prefix + file name without extension, lowercased. PostgreSQL
parses the CSV and converts the values to the column types declared on the
table: the tables must already exist and each CSV header must only contain
column names of its table (exact match, case-sensitive).
All files and tables are checked before the first load. Then files are loaded
one after the other, one transaction each: if a load fails, the tables already
loaded stay committed.
Connection settings: either give all of --postgres-host, --postgres-db,
--postgres-schema and --postgres-user (with the password), or give none of
them to use the CoAL configuration file (CONFIG_FILE_PATH, or
/mnt/coal/coal-config.toml in the cluster), then the POSTGRES_* environment
variables.
With --add-id, the transaction-local setting csm.run_id is set from --run-id,
so that a table column csm_run_id text NOT NULL DEFAULT
current_setting('csm.run_id') is filled for every row.
The postgresql user must have USAGE granted on the schema and INSERT on the
tables.
╭─ OPTIONS ────────────────────────────────────────────────────────────────────╮
│ * --csv-folder PATH Folder containing the CSV │
│ files to load, one table per │
│ .csv file │
│ ENV: CSM_OUTPUT_ABSOLUTE_PATH │
│ [required] │
│ --table-prefix PREFIX Prefix added to the file names │
│ to get the table names, │
│ lowercased (default │
│ Cosmotech_) │
│ --add-id Fill the csm_run_id column of │
│ the table with the run id │
│ --run-id TEXT Run id used with --add-id │
│ ENV: CSM_RUN_ID │
│ --postgres-host TEXT PostgreSQL host URI │
│ ENV: POSTGRES_HOST_URI │
│ --postgres-port INTEGER PostgreSQL database port │
│ ENV: POSTGRES_HOST_PORT │
│ --postgres-db TEXT PostgreSQL database name │
│ ENV: POSTGRES_DB_NAME │
│ --postgres-schema TEXT PostgreSQL schema name │
│ ENV: POSTGRES_DB_SCHEMA │
│ --postgres-user TEXT PostgreSQL connection user │
│ name │
│ ENV: POSTGRES_USER_NAME │
│ --postgres-password TEXT PostgreSQL connection password │
│ ENV: POSTGRES_USER_PASSWORD │
│ --encode-password/--no-encode… Force encoding of password to │
│ percent encoding │
│ ENV: │
│ CSM_PSQL_FORCE_PASSWORD_ENCOD… │
│ DEFAULT: encode-password │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
Optional dependency
This command needs psycopg: pip install "cosmotech-acceleration-library[custom]".