cosmotech.coal.postgresql

cosmotech.coal.postgresql.runner

PostgreSQL runner operations module.

This module provides functions for interacting with PostgreSQL databases for runner metadata operations.

Functions:

Name	Description
send_runner_metadata_to_postgresql	Send runner metadata to a PostgreSQL database.
remove_runner_metadata_from_postgresql	Removes run_id from metadata table that trigger cascade delete on other tables

send_runner_metadata_to_postgresql(configuration)

Send runner metadata to a PostgreSQL database.

Parameters:

Name	Type	Description	Default
configuration	Configuration	coal configuration	required
organization_id		Organization ID	required
workspace_id		Workspace ID	required
runner_id		Runner ID	required

remove_runner_metadata_from_postgresql(configuration)

Removes run_id from metadata table that trigger cascade delete on other tables

Parameters:

Name	Type	Description	Default
configuration	Configuration	coal configuration	required
organization_id		Organization ID	required
workspace_id		Workspace ID	required
runner_id		Runner ID	required

cosmotech.coal.postgresql.store

PostgreSQL store operations module.

This module provides functions for interacting with PostgreSQL databases for store operations.

Functions:

Name	Description
dump_store_to_postgresql	Dump Store data to a PostgreSQL database.
dump_store_to_postgresql_from_conf	Dump Store data to a PostgreSQL database.

dump_store_to_postgresql(store_folder, postgres_host, postgres_port, postgres_db, postgres_schema, postgres_user, postgres_password, table_prefix='Cosmotech_', replace=True, force_encode=False, selected_tables=[], fk_id=None)

Dump Store data to a PostgreSQL database.

Parameters:

Name	Type	Description	Default
store_folder	str	Folder containing the Store	required
postgres_host	str	PostgreSQL host	required
postgres_port	int	PostgreSQL port	required
postgres_db	str	PostgreSQL database name	required
postgres_schema	str	PostgreSQL schema	required
postgres_user	str	PostgreSQL username	required
postgres_password	str	PostgreSQL password	required
table_prefix	str	Table prefix	'Cosmotech_'
replace	bool	Whether to replace existing tables	True
force_encode	bool	force password encoding to percent encoding	False
selected_tables	list[str]	list of tables to send	[]
fk_id	str	foreign key id to add to all table on all rows	None

dump_store_to_postgresql_from_conf(configuration, replace=True, selected_tables=[], fk_id=None)

Dump Store data to a PostgreSQL database.

Parameters:

Name	Type	Description	Default
configuration	Configuration	coal Configuration	required
replace	bool	Whether to replace existing tables	True
selected_tables	list[str]	list of tables to send	[]
fk_id	str	foreign key id to add to all table on all rows	None

cosmotech.coal.postgresql.utils.PostgresUtils

Methods:

Name	Description
get_postgresql_table_schema	Get the schema of an existing PostgreSQL table using SQL queries.
send_pyarrow_table_to_postgresql
add_fk_constraint
is_metadata_exists

Attributes:

Name	Type	Description
table_prefix
db_name
db_schema
host_uri
host_port
user_name
user_password
password_encoding
full_uri	str
metadata_table_name	str

table_prefix property

db_name property

db_schema property

host_uri property

host_port property

user_name property

user_password property

password_encoding property

full_uri property

metadata_table_name property

get_postgresql_table_schema(target_table_name)

Get the schema of an existing PostgreSQL table using SQL queries.

Parameters:

Name	Type	Description	Default
target_table_name	str	Name of the table	required

Returns:

Type	Description
Optional[Schema]	PyArrow Schema if table exists, None otherwise

send_pyarrow_table_to_postgresql(data, target_table_name, replace)

add_fk_constraint(from_table, from_col, to_table, to_col)

is_metadata_exists()

cosmotech.coal.postgresql.utils

Functions:

Name	Description
adapt_table_to_schema	Adapt a PyArrow table to match a target schema with detailed logging.

adapt_table_to_schema(data, target_schema)

Adapt a PyArrow table to match a target schema with detailed logging.