Index

B

batch_read_snapshot_data_file() (dbsnaplake.project.Project method) (in module dbsnaplake.snapshot_to_staging)

C

configure_s3_write_options() (in module dbsnaplake.polars_utils) configure_s3path() (in module dbsnaplake.polars_utils)

connect_dynamodb() (dbsnaplake.project.Project method) count_records() (in module dbsnaplake.validate_datalake) create_orm_model() (in module dbsnaplake.tracker)

D

db_snapshot_manifest_file (dbsnaplake.project.Project property) dbsnaplake module dbsnaplake._import_utils module dbsnaplake.api module dbsnaplake.compaction module dbsnaplake.constants module dbsnaplake.logger module dbsnaplake.partition module dbsnaplake.polars_utils module

dbsnaplake.project module dbsnaplake.s3_loc module dbsnaplake.snapshot_to_staging module dbsnaplake.staging_to_datalake module dbsnaplake.tracker module dbsnaplake.typehint module dbsnaplake.utils module dbsnaplake.validate_datalake module DBSnapshotFileGroupManifestFile (class in dbsnaplake.snapshot_to_staging) DBSnapshotManifestFile (class in dbsnaplake.snapshot_to_staging)

E

encode_hive_partition() (in module dbsnaplake.partition)

extract_partition_data() (in module dbsnaplake.partition) extract_s3_directory() (in module dbsnaplake.staging_to_datalake)

F

from_uri() (dbsnaplake.partition.Partition class method)

G

get_merged_schema() (in module dbsnaplake.compaction) get_partitions() (in module dbsnaplake.partition) get_partitions_v2() (in module dbsnaplake.partition)

get_s3dir_datalake_partition() (dbsnaplake.s3_loc.S3Location method) get_s3dir_partition() (in module dbsnaplake.partition) get_s3dir_staging_datalake_partition() (dbsnaplake.s3_loc.S3Location method) group_by_partition() (in module dbsnaplake.polars_utils)

H

harmonize_schemas() (in module dbsnaplake.compaction)

I

iter_staging_datalake_partition() (dbsnaplake.s3_loc.S3Location method)

L

list_parquet_files() (dbsnaplake.partition.Partition method)

M

merge_dataframes() (in module dbsnaplake.compaction) module dbsnaplake dbsnaplake._import_utils dbsnaplake.api dbsnaplake.compaction dbsnaplake.constants dbsnaplake.logger dbsnaplake.partition dbsnaplake.polars_utils dbsnaplake.project dbsnaplake.s3_loc dbsnaplake.snapshot_to_staging dbsnaplake.staging_to_datalake dbsnaplake.tracker dbsnaplake.typehint dbsnaplake.utils dbsnaplake.validate_datalake

P

Partition (class in dbsnaplake.partition) (class in dbsnaplake.validate_datalake) PartitionFileGroupManifestFile (class in dbsnaplake.staging_to_datalake) plan_partition_compaction() (dbsnaplake.staging_to_datalake.PartitionFileGroupManifestFile class method)

print_manifest_file_info() (in module dbsnaplake.project) process_db_snapshot_file_group_manifest_file() (in module dbsnaplake.snapshot_to_staging) process_partition_file_group_manifest_file() (in module dbsnaplake.staging_to_datalake) Project (class in dbsnaplake.project)

R

read_all_groups() (dbsnaplake.snapshot_to_staging.DBSnapshotFileGroupManifestFile class method) (dbsnaplake.staging_to_datalake.PartitionFileGroupManifestFile class method)

read_many_parquet_from_s3() (in module dbsnaplake.polars_utils) read_parquet_from_s3() (in module dbsnaplake.polars_utils) repr_data_size() (in module dbsnaplake.utils)

S

s3_loc (dbsnaplake.project.Project property) s3dir (dbsnaplake.partition.Partition property) s3dir_datalake (dbsnaplake.s3_loc.S3Location property) s3dir_partition_file_group_manifest (dbsnaplake.s3_loc.S3Location property) s3dir_partition_file_group_manifest_data (dbsnaplake.s3_loc.S3Location property) s3dir_partition_file_group_manifest_summary (dbsnaplake.s3_loc.S3Location property) s3dir_snapshot_file_group_manifest (dbsnaplake.s3_loc.S3Location property) s3dir_snapshot_file_group_manifest_data (dbsnaplake.s3_loc.S3Location property) s3dir_snapshot_file_group_manifest_summary (dbsnaplake.s3_loc.S3Location property) s3dir_staging (dbsnaplake.s3_loc.S3Location property)

s3dir_staging_datalake (dbsnaplake.s3_loc.S3Location property) s3dir_staging_file_group_manifest (dbsnaplake.s3_loc.S3Location property) s3dir_staging_file_group_manifest_data (dbsnaplake.s3_loc.S3Location property) s3dir_staging_file_group_manifest_summary (dbsnaplake.s3_loc.S3Location property) s3dir_staging_manifest (dbsnaplake.s3_loc.S3Location property) S3Location (class in dbsnaplake.s3_loc) s3path_validate_datalake_result (dbsnaplake.s3_loc.S3Location property) split_into_groups() (dbsnaplake.snapshot_to_staging.DBSnapshotManifestFile method) StagingFileGroupManifestFile (class in dbsnaplake.snapshot_to_staging) StatusEnum (class in dbsnaplake.tracker)

T

task_model_step_0_prepare_db_snapshot_manifest (dbsnaplake.project.Project property) task_model_step_1_1_plan_snapshot_to_staging (dbsnaplake.project.Project property)

task_model_step_1_2_process_db_snapshot_file_group_manifest_file (dbsnaplake.project.Project property) task_model_step_2_1_plan_staging_to_datalake (dbsnaplake.project.Project property) task_model_step_2_2_process_partition_file_group_manifest_file (dbsnaplake.project.Project property)

V

validate_datalake() (in module dbsnaplake.validate_datalake)

ValidateDatalakeResult (class in dbsnaplake.validate_datalake)

W

write_to_s3() (in module dbsnaplake.polars_utils)