Skip to content

load-csv-folder

Help command

> csm-data custom load-csv-folder --help

 Usage: csm-data custom load-csv-folder [OPTIONS]                               

 Stream every CSV file of a folder into its own existing PostgreSQL table with  
 COPY FROM STDIN                                                                
 Each .csv file directly in the folder (not recursive) is loaded into the table 
 named table prefix + file name without extension, lowercased. PostgreSQL       
 parses the CSV and converts the values to the column types declared on the     
 table: the tables must already exist and each CSV header must only contain     
 column names of its table (exact match, case-sensitive).                       

 All files and tables are checked before the first load. Then files are loaded  
 one after the other, one transaction each: if a load fails, the tables already 
 loaded stay committed.                                                         

 Connection settings: either give all of --postgres-host, --postgres-db,        
 --postgres-schema and --postgres-user (with the password), or give none of     
 them to use the CoAL configuration file (CONFIG_FILE_PATH, or                  
 /mnt/coal/coal-config.toml in the cluster), then the POSTGRES_* environment    
 variables.                                                                     

 With --add-id, the transaction-local setting csm.run_id is set from --run-id,  
 so that a table column csm_run_id text NOT NULL DEFAULT                        
 current_setting('csm.run_id') is filled for every row.                         

 The postgresql user must have USAGE granted on the schema and INSERT on the    
 tables.                                                                        

╭─ OPTIONS ────────────────────────────────────────────────────────────────────╮
│ *  --csv-folder                      PATH     Folder containing the CSV      │
│                                               files to load, one table per   │
│                                               .csv file                      │
│                                               ENV: CSM_OUTPUT_ABSOLUTE_PATH  │
│                                               [required]                     │
│    --table-prefix                    PREFIX   Prefix added to the file names │
│                                               to get the table names,        │
│                                               lowercased (default            │
│                                               Cosmotech_)                    │
│    --add-id                                   Fill the csm_run_id column of  │
│                                               the table with the run id      │
│    --run-id                          TEXT     Run id used with --add-id      │
│                                               ENV: CSM_RUN_ID                │
│    --postgres-host                   TEXT     PostgreSQL host URI            │
│                                               ENV: POSTGRES_HOST_URI         │
│    --postgres-port                   INTEGER  PostgreSQL database port       │
│                                               ENV: POSTGRES_HOST_PORT        │
│    --postgres-db                     TEXT     PostgreSQL database name       │
│                                               ENV: POSTGRES_DB_NAME          │
│    --postgres-schema                 TEXT     PostgreSQL schema name         │
│                                               ENV: POSTGRES_DB_SCHEMA        │
│    --postgres-user                   TEXT     PostgreSQL connection user     │
│                                               name                           │
│                                               ENV: POSTGRES_USER_NAME        │
│    --postgres-password               TEXT     PostgreSQL connection password │
│                                               ENV: POSTGRES_USER_PASSWORD    │
│    --encode-password/--no-encode…             Force encoding of password to  │
│                                               percent encoding               │
│                                               ENV:                           │
│                                               CSM_PSQL_FORCE_PASSWORD_ENCOD… │
│                                               DEFAULT: encode-password       │
│    --help                                     Show this message and exit.    │
╰──────────────────────────────────────────────────────────────────────────────╯

Optional dependency

This command needs psycopg: pip install "cosmotech-acceleration-library[custom]".