cosmotech.coal.store¶

`cosmotech.coal.store.csv` ¶

Functions:

Name	Description
`store_csv_file`
`convert_store_table_to_csv`

`store_csv_file(table_name, csv_path, replace_existsing_file=False, store=None)` ¶

`convert_store_table_to_csv(table_name, csv_path, replace_existsing_file=False, store=None)` ¶

`cosmotech.coal.store.native_python` ¶

Functions:

Name	Description
`store_pylist`
`convert_table_as_pylist`

`store_pylist(table_name, data, replace_existsing_file=False, store=None)` ¶

`convert_table_as_pylist(table_name, store=None)` ¶

`cosmotech.coal.store.output.aws_channel.AwsChannel` ¶

Bases: ChannelInterface


              flowchart TD
              cosmotech.coal.store.output.aws_channel.AwsChannel[AwsChannel]
              cosmotech.coal.store.output.channel_interface.ChannelInterface[ChannelInterface]

                              cosmotech.coal.store.output.channel_interface.ChannelInterface --> cosmotech.coal.store.output.aws_channel.AwsChannel
                


              click cosmotech.coal.store.output.aws_channel.AwsChannel href "" "cosmotech.coal.store.output.aws_channel.AwsChannel"
              click cosmotech.coal.store.output.channel_interface.ChannelInterface href "" "cosmotech.coal.store.output.channel_interface.ChannelInterface"

Output channel that sends the store's tables or database file to an S3 bucket.

Methods:

Name	Description
`send`	Upload the store's database file or tables to the configured S3 bucket.
`delete`	Delete all objects previously uploaded to the configured S3 bucket.

`send(filter=None)` ¶

Upload the store's database file or tables to the configured S3 bucket.

Parameters:

Name	Type	Description	Default
`filter`	`Optional[list[str]]`	Optional list of table names to restrict the upload to	`None`

Returns:

Type	Description
`bool`	True if the data was uploaded successfully

Raises:

Type	Description
`ValueError`	If the configured output type is not sqlite, csv or parquet

`delete()` ¶

Delete all objects previously uploaded to the configured S3 bucket.

`cosmotech.coal.store.output.az_storage_channel.AzureStorageChannel` ¶

Bases: ChannelInterface


              flowchart TD
              cosmotech.coal.store.output.az_storage_channel.AzureStorageChannel[AzureStorageChannel]
              cosmotech.coal.store.output.channel_interface.ChannelInterface[ChannelInterface]

                              cosmotech.coal.store.output.channel_interface.ChannelInterface --> cosmotech.coal.store.output.az_storage_channel.AzureStorageChannel
                


              click cosmotech.coal.store.output.az_storage_channel.AzureStorageChannel href "" "cosmotech.coal.store.output.az_storage_channel.AzureStorageChannel"
              click cosmotech.coal.store.output.channel_interface.ChannelInterface href "" "cosmotech.coal.store.output.channel_interface.ChannelInterface"

Output channel that sends the store's tables to an Azure Blob Storage container.

Methods:

Name	Description
`send`	Dump the store's tables to the configured Azure Blob Storage container.
`delete`	Delete all blobs previously uploaded to the configured Azure Blob Storage container.

`send(filter=None)` ¶

Dump the store's tables to the configured Azure Blob Storage container.

Parameters:

Name	Type	Description	Default
`filter`	`Optional[list[str]]`	Optional list of table names to restrict the dump to	`None`

Returns:

Type	Description
`bool`	True if the data was uploaded successfully

`delete()` ¶

Delete all blobs previously uploaded to the configured Azure Blob Storage container.

`cosmotech.coal.store.output.channel_interface.ChannelInterface` ¶

Base class for an output channel that can send the store's data to a target and delete it.

Methods:

Name	Description
`send`
`delete`
`is_available`	Check whether all required configuration keys are present, falling back to global config.

`send(filter=None)` ¶

`delete()` ¶

`is_available()` ¶

Check whether all required configuration keys are present, falling back to global config.

Returns:

Type	Description
`bool`	True if the channel is fully configured and can be used

`cosmotech.coal.store.output.channel_spliter.ChannelSpliter` ¶

Bases: ChannelInterface


              flowchart TD
              cosmotech.coal.store.output.channel_spliter.ChannelSpliter[ChannelSpliter]
              cosmotech.coal.store.output.channel_interface.ChannelInterface[ChannelInterface]

                              cosmotech.coal.store.output.channel_interface.ChannelInterface --> cosmotech.coal.store.output.channel_spliter.ChannelSpliter
                


              click cosmotech.coal.store.output.channel_spliter.ChannelSpliter href "" "cosmotech.coal.store.output.channel_spliter.ChannelSpliter"
              click cosmotech.coal.store.output.channel_interface.ChannelInterface href "" "cosmotech.coal.store.output.channel_interface.ChannelInterface"

Channel that fans out the store's data to every configured output channel.

Methods:

Name	Description
`send`	Send the store's data through every available target channel.
`delete`	Delete the data previously sent through every available target channel.

`send(filter=None)` ¶

Send the store's data through every available target channel.

Parameters:

Name	Type	Description	Default
`filter`	`Optional[list[str]]`	Optional list of table names to restrict the send to	`None`

Returns:

Type	Description
`bool`	True if at least one target channel sent its data successfully

`delete()` ¶

Delete the data previously sent through every available target channel.

Returns:

Type	Description
`bool`	True if at least one target channel deleted its data successfully

`cosmotech.coal.store.output.postgres_channel.PostgresChannel` ¶

Bases: ChannelInterface


              flowchart TD
              cosmotech.coal.store.output.postgres_channel.PostgresChannel[PostgresChannel]
              cosmotech.coal.store.output.channel_interface.ChannelInterface[ChannelInterface]

                              cosmotech.coal.store.output.channel_interface.ChannelInterface --> cosmotech.coal.store.output.postgres_channel.PostgresChannel
                


              click cosmotech.coal.store.output.postgres_channel.PostgresChannel href "" "cosmotech.coal.store.output.postgres_channel.PostgresChannel"
              click cosmotech.coal.store.output.channel_interface.ChannelInterface href "" "cosmotech.coal.store.output.channel_interface.ChannelInterface"

Output channel that sends the store's tables and runner metadata to a PostgreSQL database.

Methods:

Name	Description
`send`	Send the runner's metadata and the store's tables to the configured PostgreSQL database.
`delete`	Delete the runner's metadata, cascading to delete its associated data in PostgreSQL.

`send(filter=None)` ¶

Send the runner's metadata and the store's tables to the configured PostgreSQL database.

Parameters:

Name	Type	Description	Default
`filter`	`Optional[list[str]]`	Optional list of table names to restrict the send to	`None`

Returns:

Type	Description
`bool`	True if the data was sent successfully

`delete()` ¶

Delete the runner's metadata, cascading to delete its associated data in PostgreSQL.

`cosmotech.coal.store.pandas` ¶

Functions:

Name	Description
`store_dataframe`
`convert_store_table_to_dataframe`

`store_dataframe(table_name, dataframe, replace_existsing_file=False, store=None)` ¶

`convert_store_table_to_dataframe(table_name, store=None)` ¶

`cosmotech.coal.store.parquet` ¶

Functions:

Name	Description
`store_parquet_file`
`convert_store_table_to_parquet`

`store_parquet_file(table_name, parquet_path, replace_existsing_file=False, store=None)` ¶

`convert_store_table_to_parquet(table_name, parquet_path, replace_existsing_file=False, store=None)` ¶

`cosmotech.coal.store.pyarrow` ¶

Functions:

Name	Description
`store_table`
`convert_store_table_to_dataframe`

`store_table(table_name, data, replace_existsing_file=False, store=None)` ¶

`convert_store_table_to_dataframe(table_name, store=None)` ¶

`cosmotech.coal.store.store` ¶

Functions:

Name	Description
`table_name_to_lower`

`table_name_to_lower(func)` ¶

`cosmotech.coal.store.store.Store` ¶

Methods:

Name	Description
`sanitize_column`
`reset`
`get_table`
`table_exists`
`get_table_schema`
`add_table`
`execute_query`
`list_tables`

cosmotech.coal.store¶

cosmotech.coal.store.csv ¶

store_csv_file(table_name, csv_path, replace_existsing_file=False, store=None) ¶

convert_store_table_to_csv(table_name, csv_path, replace_existsing_file=False, store=None) ¶

cosmotech.coal.store.native_python ¶

store_pylist(table_name, data, replace_existsing_file=False, store=None) ¶

convert_table_as_pylist(table_name, store=None) ¶

cosmotech.coal.store.output.aws_channel.AwsChannel ¶

send(filter=None) ¶

delete() ¶

cosmotech.coal.store.output.az_storage_channel.AzureStorageChannel ¶

send(filter=None) ¶

delete() ¶

cosmotech.coal.store.output.channel_interface.ChannelInterface ¶

send(filter=None) ¶

delete() ¶

is_available() ¶

cosmotech.coal.store.output.channel_spliter.ChannelSpliter ¶

send(filter=None) ¶

delete() ¶

cosmotech.coal.store.output.postgres_channel.PostgresChannel ¶

send(filter=None) ¶

delete() ¶

cosmotech.coal.store.pandas ¶

store_dataframe(table_name, dataframe, replace_existsing_file=False, store=None) ¶

convert_store_table_to_dataframe(table_name, store=None) ¶

cosmotech.coal.store.parquet ¶

store_parquet_file(table_name, parquet_path, replace_existsing_file=False, store=None) ¶

convert_store_table_to_parquet(table_name, parquet_path, replace_existsing_file=False, store=None) ¶

cosmotech.coal.store.pyarrow ¶

store_table(table_name, data, replace_existsing_file=False, store=None) ¶

convert_store_table_to_dataframe(table_name, store=None) ¶

cosmotech.coal.store.store ¶

table_name_to_lower(func) ¶

cosmotech.coal.store.store.Store ¶

sanitize_column(column_name) staticmethod ¶

reset() ¶

get_table(table_name) ¶

table_exists(table_name) ¶

get_table_schema(table_name) ¶

add_table(table_name, data=pyarrow.Table, replace=False) ¶

execute_query(sql_query, parameters=None) ¶

list_tables() ¶

`cosmotech.coal.store.csv` ¶

`store_csv_file(table_name, csv_path, replace_existsing_file=False, store=None)` ¶

`convert_store_table_to_csv(table_name, csv_path, replace_existsing_file=False, store=None)` ¶

`cosmotech.coal.store.native_python` ¶

`store_pylist(table_name, data, replace_existsing_file=False, store=None)` ¶

`convert_table_as_pylist(table_name, store=None)` ¶

`cosmotech.coal.store.output.aws_channel.AwsChannel` ¶

`send(filter=None)` ¶

`delete()` ¶

`cosmotech.coal.store.output.az_storage_channel.AzureStorageChannel` ¶

`send(filter=None)` ¶

`delete()` ¶

`cosmotech.coal.store.output.channel_interface.ChannelInterface` ¶

`send(filter=None)` ¶

`delete()` ¶

`is_available()` ¶

`cosmotech.coal.store.output.channel_spliter.ChannelSpliter` ¶

`send(filter=None)` ¶

`delete()` ¶

`cosmotech.coal.store.output.postgres_channel.PostgresChannel` ¶

`send(filter=None)` ¶

`delete()` ¶

`cosmotech.coal.store.pandas` ¶

`store_dataframe(table_name, dataframe, replace_existsing_file=False, store=None)` ¶

`convert_store_table_to_dataframe(table_name, store=None)` ¶

`cosmotech.coal.store.parquet` ¶

`store_parquet_file(table_name, parquet_path, replace_existsing_file=False, store=None)` ¶

`convert_store_table_to_parquet(table_name, parquet_path, replace_existsing_file=False, store=None)` ¶

`cosmotech.coal.store.pyarrow` ¶

`store_table(table_name, data, replace_existsing_file=False, store=None)` ¶

`convert_store_table_to_dataframe(table_name, store=None)` ¶

`cosmotech.coal.store.store` ¶

`table_name_to_lower(func)` ¶

`cosmotech.coal.store.store.Store` ¶

`sanitize_column(column_name)` `staticmethod` ¶

`reset()` ¶

`get_table(table_name)` ¶

`table_exists(table_name)` ¶

`get_table_schema(table_name)` ¶

`add_table(table_name, data=pyarrow.Table, replace=False)` ¶

`execute_query(sql_query, parameters=None)` ¶

`list_tables()` ¶