q-Ph H JddlmZddlZddlmZddlmZddlmZmZddl m Z ddl m Z ddl mZmZdd lmZdd lmZmZdd lmZdd lmZdd lmZerddlmZddlmZddlmZmZm Z ddl!m"Z"dddddddddd9d'Z#dddddddd(d:d*Z$d+d,d;d0Z% dd5Z'd?d8Z(dS)@) annotationsN)datetime)Path) TYPE_CHECKINGAny)urlparse) from_arrow)NullTime) unpack_dtypes)_DELTALAKE_AVAILABLE deltalake) scan_parquet)scan_pyarrow_dataset)Schema)Literal DeltaTable) DataFrameDataType LazyFrame)CredentialProviderFunctionautoF)versioncolumnsrechunkstorage_optionscredential_providerdelta_table_options use_pyarrowpyarrow_optionssourcestr | DeltaTablerint | str | datetime | Nonerlist[str] | Noner bool | Nonerdict[str, Any] | Noner3CredentialProviderFunction | Literal['auto'] | Nonerr boolr!returnrc t||||||||} || |} | S)a7 Reads into a DataFrame from a Delta lake table. Parameters ---------- source DeltaTable or a Path or URI to the root of the Delta lake table. Note: For Local filesystem, absolute and relative paths are supported but for the supported object storages - GCS, Azure and S3 full URI must be provided. version Numerical version or timestamp version of the Delta lake table. Note: If `version` is not provided, the latest version of delta lake table is read. columns Columns to select. Accepts a list of column names. rechunk Make sure that all columns are contiguous in memory by aggregating the chunks into a single array. storage_options Extra options for the storage backends supported by `deltalake`. For cloud storages, this may include configurations for authentication etc. More info is available `here `__. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. delta_table_options Additional keyword arguments while reading a Delta lake Table. use_pyarrow Flag to enable pyarrow dataset reads. pyarrow_options Keyword arguments while converting a Delta lake Table to pyarrow table. Returns ------- DataFrame Examples -------- Reads a Delta table from local filesystem. Note: Since version is not provided, the latest version of the delta table is read. >>> table_path = "/path/to/delta-table/" >>> pl.read_delta(table_path) # doctest: +SKIP Reads a specific version of the Delta table from local filesystem. Note: This will fail if the provided version of the delta table does not exist. >>> pl.read_delta(table_path, version=1) # doctest: +SKIP Time travel a delta table from local filesystem using a timestamp version. >>> pl.read_delta( ... table_path, version=datetime(2020, 1, 1, tzinfo=timezone.utc) ... ) # doctest: +SKIP Reads a Delta table from AWS S3. See a list of supported storage options for S3 `here `__. >>> table_path = "s3://bucket/path/to/delta-table/" >>> storage_options = { ... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID", ... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY", ... } >>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP Reads a Delta table from Google Cloud storage (GCS). See a list of supported storage options for GCS `here `__. >>> table_path = "gs://bucket/path/to/delta-table/" >>> storage_options = {"SERVICE_ACCOUNT": "SERVICE_ACCOUNT_JSON_ABSOLUTE_PATH"} >>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP Reads a Delta table from Azure. Following type of table paths are supported, * az:/// * adl:/// * abfs:/// See a list of supported storage options for Azure `here `__. >>> table_path = "az://container/path/to/delta-table/" >>> storage_options = { ... "AZURE_STORAGE_ACCOUNT_NAME": "AZURE_STORAGE_ACCOUNT_NAME", ... "AZURE_STORAGE_ACCOUNT_KEY": "AZURE_STORAGE_ACCOUNT_KEY", ... } >>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP Reads a Delta table with additional delta specific options. In the below example, `without_files` option is used which loads the table without file tracking information. >>> table_path = "/path/to/delta-table/" >>> delta_table_options = {"without_files": True} >>> pl.read_delta( ... table_path, delta_table_options=delta_table_options ... ) # doctest: +SKIP )r"rrrrr r!r) scan_deltaselectcollect) r"rrrrrrr r!dfs O/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/polars/io/delta.py read_deltar1sXv '//'   B YYw   ::<<)rrrrr r!rrc ti}ddlm} ddlm} ddlm} t|| s| |||d} n||dkrd} t| d} ~| r!| x}r | |}t|||| i|pi|nd| }|r |pi}|j d i|}t|S|d } t| ddl }dd lm}dd lm}m}m}|}|j|ks |j|krd |jd|d|} || |jdkrG|j@h|j|}t1|dkrd|d} || |d}t7|jg|j}|j}d!d}|||\}}| }|j!"dr d|D}tG||t1|dkr|nddt1|dk|| |pdS)"aY Lazily read from a Delta lake table. Parameters ---------- source DeltaTable or a Path or URI to the root of the Delta lake table. Note: For Local filesystem, absolute and relative paths are supported but for the supported object storages - GCS, Azure and S3 full URI must be provided. version Numerical version or timestamp version of the Delta lake table. Note: If `version` is not provided, the latest version of delta lake table is read. storage_options Extra options for the storage backends supported by `deltalake`. For cloud storages, this may include configurations for authentication etc. More info is available `here `__. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. delta_table_options Additional keyword arguments while reading a Delta lake Table. use_pyarrow Flag to enable pyarrow dataset reads. pyarrow_options Keyword arguments while converting a Delta lake Table to pyarrow table. Use this parameter when filtering on partitioned columns or to read from a 'fsspec' supported filesystem. rechunk Make sure that all columns are contiguous in memory by aggregating the chunks into a single array. Returns ------- LazyFrame Examples -------- Creates a scan for a Delta table from local filesystem. Note: Since version is not provided, the latest version of the delta table is read. >>> table_path = "/path/to/delta-table/" >>> pl.scan_delta(table_path).collect() # doctest: +SKIP Creates a scan for a specific version of the Delta table from local filesystem. Note: This will fail if the provided version of the delta table does not exist. >>> pl.scan_delta(table_path, version=1).collect() # doctest: +SKIP Time travel a delta table from local filesystem using a timestamp version. >>> pl.scan_delta( ... table_path, version=datetime(2020, 1, 1, tzinfo=timezone.utc) ... ).collect() # doctest: +SKIP Creates a scan for a Delta table from AWS S3. See a list of supported storage options for S3 `here `__. >>> table_path = "s3://bucket/path/to/delta-table/" >>> storage_options = { ... "AWS_REGION": "eu-central-1", ... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID", ... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY", ... } >>> pl.scan_delta( ... table_path, storage_options=storage_options ... ).collect() # doctest: +SKIP Creates a scan for a Delta table from Google Cloud storage (GCS). See a list of supported storage options for GCS `here `__. >>> table_path = "gs://bucket/path/to/delta-table/" >>> storage_options = {"SERVICE_ACCOUNT": "SERVICE_ACCOUNT_JSON_ABSOLUTE_PATH"} >>> pl.scan_delta( ... table_path, storage_options=storage_options ... ).collect() # doctest: +SKIP Creates a scan for a Delta table from Azure. Supported options for Azure are available `here `__. Following type of table paths are supported, * az:/// * adl:/// * abfs[s]:/// >>> table_path = "az://container/path/to/delta-table/" >>> storage_options = { ... "AZURE_STORAGE_ACCOUNT_NAME": "AZURE_STORAGE_ACCOUNT_NAME", ... "AZURE_STORAGE_ACCOUNT_KEY": "AZURE_STORAGE_ACCOUNT_KEY", ... } >>> pl.scan_delta( ... table_path, storage_options=storage_options ... ).collect() # doctest: +SKIP Creates a scan for a Delta table with additional delta specific options. In the below example, `without_files` option is used which loads the table without file tracking information. >>> table_path = "/path/to/delta-table/" >>> delta_table_options = {"without_files": True} >>> pl.scan_delta( ... table_path, delta_table_options=delta_table_options ... ).collect() # doctest: +SKIP rr)!_init_credential_provider_builder)+_get_credentials_from_provider_expiry_awarer,Nrz?cannot use credential_provider when passing a DeltaTable object) table_pathrrrz7To make use of pyarrow_options, set use_pyarrow to True)DeltaProtocolError)MAX_SUPPORTED_READER_VERSIONNOT_SUPPORTED_READER_VERSIONSUPPORTED_READER_FEATURESz&The table's minimum reader version is z5 but polars delta scanner only supports version 1 or z with these reader features: z)The table has set these reader features: z= but these are not yet supported by the polars delta scanner.T)as_large_typesschemarpartition_columns list[str]r*tuple[Schema, Schema]c(t|dkr|tgfSg}g}|D]8\}}||vr|||f!|||f9t|t|fS)Nr)lenritemsappend)r=r> main_schema hive_schemanamedtypes r0 _split_schemaz!scan_delta.._split_schema|s  ! !Q & &6"::% %  !<<>> 2 2KD%(((""D%=1111""D%=1111k""F;$7$777r2 lakefs://c:g|]}|ddS)rJzs3://)replace).0file_uris r0 zscan_delta..s(VVVX%%k7;;VVVr2F)r=rFallow_missing_columnshive_partitioningrrr)r=rr>r?r*r@)$_check_if_delta_availablerr,polars.io.cloud.credential_provider._builderr4.polars.io.cloud.credential_provider._providersr5 isinstance ValueErrorbuild_credential_provider_get_delta_lake_tableto_pyarrow_datasetrpyarrowdeltalake.exceptionsr7deltalake.tabler8r9r:protocolmin_reader_versionreader_features differencerBr= to_pyarrowr Table from_pylistmetadatar> file_uris table_uri startswithr)r"rrrrr r!rcredential_provider_credsrr4r5credential_provider_buildermsgproviderdl_tblpa_dspar7r8r9r:table_protocolmissing_features delta_schema polars_schemar>rIrErFrfs r0r,r,s@ "$$$$$$ fj ) )+&G&G ,' ' ##  (-@F-J-JOoo&*#" /IIKKK %P$O % % !#*.I.U E%2 D*C D D/   F+)/R))<NNNNC$$S)) )==??--T-BBLrx33B EEFFMM));8888" -}]=0.8.0. Read the documentation `here `_. NzWhen supplying a DeltaTable directly, `version`, `storage_options`, and `delta_table_options` are ignored. To silence this warning, don't supply those parameters.) stacklevel)rrr) rSrVrranywarningswarnRuntimeWarningr}rurload_as_version)r6rrrr|rms r0rYrYs,*i233 t#t+#4/    MK      " *:66L gX / / (%  +  "   %   + "   w''' Mr2Nonec6tsd}t|dS)Nz=deltalake is not installed Please run: pip install deltalake)r ModuleNotFoundError)rks r0rSrSs' 'O!#&&&''r2dtypeslist[DataType]cjt|}tth}||zx}rd|}t|dS)Nz+dataframe contains unsupported data types: )r r r TypeError)r schema_dtypesunsupported_typesoverlaprks r0_check_for_unsupported_typesrsN!6*Mt   "333wGGGGnnr2)r"r#rr$rr%rr&rr'rr(rr'r r)r!r'r*r)r"r#rr$rr'rr(rr'r r)r!r'rr&r*r)rgrurtr)r*ru)NNN) r6r#rr$rr'rr'r*r~)r*r)rrr*r)) __future__rrrpathlibrtypingrr urllib.parserpolars.convertr polars.datatypesr r polars.datatypes.convertr polars.dependenciesr rpolars.io.parquetr#polars.io.pyarrow_dataset.functionsr polars.schemarrrpolarsrrrpolars.io.cloudrr1r,r}rYrSrrRr2r0rsJ""""""%%%%%%%%!!!!!!%%%%%%''''''''222222????????******DDDDDD ;$$$$$$5555555555:::::: ,0 $-1OU15-1HHHHHH\,0-1OU15-1zzzzzzz?C      ,0-115 11111h''''      r2