q-Ph]ddlmZddlZddlZddlmZddlmZmZm Z ddl m Z ddl mZddlmZddlmZddlmZmZmZdd lmZdd lmZdd lmZdd lmZm Z dd l!m"Z"ej#e$5ddl%m&Z&ddl%m'Z(ddl%m)Z*dddn #1swxYwYer,ddlm+Z+ddl m,Z,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2ddl3m4Z4ddl5m6Z6ddl!m7Z7edddeddddddddddddddddddd dddddd!dWdCZ8ddddddDdXdFZ9dYdIZ)dZdKZ'edddeddddddddddddddddddd ddddLd[dQZ:dddddddddddddddd ddddRd\dVZ;dS)]) annotationsN)Path)IO TYPE_CHECKINGAny)concat)deprecate_renamed_parameter)issue_unstable_warning)is_int_sequenceis_path_or_str_sequencenormalize_filepath)wrap_ldf) from_arrow)import_optional)parse_row_index_argsprepare_file_arg)!_init_credential_provider_builder) PyLazyFrame)read_parquet_metadata)read_parquet_schema)Literal) DataFrameDataType LazyFrame) FileSourceParallelStrategy SchemaDict)ScanCastOptions)CredentialProviderFunction)CredentialProviderBuilderrow_count_namerow_index_namez0.20.4)versionrow_count_offsetrow_index_offsetautoTF)columnsn_rowsr"r%paralleluse_statisticshive_partitioningglobschema hive_schematry_parse_hive_datesrechunk low_memorystorage_optionscredential_providerretries use_pyarrowpyarrow_options memory_mapinclude_file_pathsallow_missing_columnssourcerr(list[int] | list[str] | Noner) int | None str | Noneintr*rr+boolr, bool | Noner-r.SchemaDict | Noner/r0r1r2r3dict[str, Any] | Noner43CredentialProviderFunction | Literal['auto'] | Noner5r6r7r8r9r:returnrcZ| d}t|| d}t||ra|d}t||d}t|| d}t|| d}t|t|||||| St |fid |d |d |d |d |d|d| d| d| d| d| ddd|d|d|d|d|d|}|Lt |r(|tj|}n||}| S)a Read into a DataFrame from a parquet file. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path(s) to a file or directory When needing to authenticate for scanning cloud locations, see the `storage_options` parameter. File-like objects are supported (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). For file-like objects, the stream position may not be updated accordingly after reading. columns Columns to select. Accepts a list of column indices (starting at zero) or a list of column names. n_rows Stop reading from parquet file after reading `n_rows`. Only valid when `use_pyarrow=False`. row_index_name Insert a row index column with the given name into the DataFrame as the first column. If set to `None` (default), no row index column is created. row_index_offset Start the row index at this offset. Cannot be negative. Only used if `row_index_name` is set. parallel : {'auto', 'columns', 'row_groups', 'none'} This determines the direction of parallelism. 'auto' will try to determine the optimal direction. use_statistics Use statistics in the parquet to determine if pages can be skipped from reading. hive_partitioning Infer statistics and schema from Hive partitioned URL and use them to prune reads. This is unset by default (i.e. `None`), meaning it is automatically enabled when a single directory is passed, and otherwise disabled. glob Expand path given via globbing rules. schema Specify the datatypes of the columns. The datatypes must match the datatypes in the file(s). If there are extra columns that are not in the file(s), consider also enabling `allow_missing_columns`. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. hive_schema The column names and data types of the columns by which the data is partitioned. If set to `None` (default), the schema of the Hive partitions is inferred. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. try_parse_hive_dates Whether to try parsing hive values as date/datetime types. rechunk Make sure that all columns are contiguous in memory by aggregating the chunks into a single array. low_memory Reduce memory pressure at the expense of performance. storage_options Options that indicate how to connect to a cloud provider. The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here: * `aws `_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. retries Number of retries if accessing a cloud instance fails. use_pyarrow Use PyArrow instead of the Rust-native Parquet reader. The PyArrow reader is more stable. pyarrow_options Keyword arguments for `pyarrow.parquet.read_table `_. memory_map Memory map underlying file. This will likely increase performance. Only used when `use_pyarrow=True`. include_file_paths Include the path of the source file(s) as a column with this name. Only valid when `use_pyarrow=False`. allow_missing_columns When reading a list of parquet files, if a column existing in the first file cannot be found in subsequent files, the default behavior is to raise an error. However, if `allow_missing_columns` is set to `True`, a full-NULL column is returned instead of erroring for the files that do not contain the column. Returns ------- DataFrame See Also -------- scan_parquet: Lazily read from a parquet file or multiple files via glob patterns. scan_pyarrow_dataset Warnings -------- Calling `read_parquet().lazy()` is an antipattern as this forces Polars to materialize a full parquet file and therefore cannot push any optimizations into the reader. Therefore always prefer `scan_parquet` if you want to work with `LazyFrame` s. Nz@the `schema` parameter of `read_parquet` is considered unstable.zEthe `hive_schema` parameter of `read_parquet` is considered unstable.z/`n_rows` cannot be used with `use_pyarrow=True`z;`include_file_paths` cannot be used with `use_pyarrow=True`z/`schema` cannot be used with `use_pyarrow=True`zwcannot use `hive_partitions` with `use_pyarrow=True` Hint: Pass `pyarrow_options` instead with a 'partitioning' entry.r(r3r7r8r1r)r"r%r*r+r,r.r/r0r1r2cacheFr3r4r5r-r9r:) r ValueError TypeError_read_parquet_with_pyarrow scan_parquetr selectFnthcollect)r;r(r)r"r%r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:msglfs [/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/polars/io/parquet/functions.py read_parquetrT+spPs###Us###  CCS// !  )OCS// !  CCS// !  "X C.. ) ++!         v &~ *)      &~  ,+ v  K 21  : e ( 0/  ! "T# $.-% &43' B, 7 # # $15>>**BB7##B ::<<rGWstr | Path | IO[bytes] | bytes | list[str] | list[Path] | list[IO[bytes]] | list[bytes]ctddd}|pi}g}t|trAt|dkr*t|dtt jfr|}n|g}n|g}g}|D]c}t|d|5} |j| f||d|} dddn #1swxYwYt| | } | | dt|d kr|dSt|S) Nzpyarrow.parquetz`_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. retries Number of retries if accessing a cloud instance fails. include_file_paths Include the path of the source file(s) as a column with this name. allow_missing_columns When reading a list of parquet files, if a column existing in the first file cannot be found in subsequent files, the default behavior is to raise an error. However, if `allow_missing_columns` is set to `True`, a full-NULL column is returned instead of erroring for the files that do not contain the column. cast_options Configuration for column type-casting during scans. Useful for datasets containing files that have differing schemas. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. See Also -------- read_parquet scan_pyarrow_dataset Examples -------- Scan a local Parquet file. >>> pl.scan_parquet("path/to/file.parquet") # doctest: +SKIP Scan a file on AWS S3. >>> source = "s3://bucket/*.parquet" >>> pl.scan_parquet(source) # doctest: +SKIP >>> storage_options = { ... "aws_access_key_id": "", ... "aws_secret_access_key": "", ... "aws_region": "us-east-1", ... } >>> pl.scan_parquet(source, storage_options=storage_options) # doctest: +SKIP Nz@the `schema` parameter of `scan_parquet` is considered unstable.zEthe `hive_schema` parameter of `scan_parquet` is considered unstable.zFThe `cast_options` parameter of `scan_parquet` is considered unstable.Frnc0g|]}t|dS)Frn)r ).0r;s rS z scan_parquet..s3   FL v5 A A A   rUrLr)rHr*r1r"r%r3r4r2r+r,r.r/r0r5r-r9r:rv)r r\rprr r r_scan_parquet_impl)r;r)r"r%r*r+r,r-r.r/r0r1r2rHr3r4r5r9r:rvrQcredential_provider_builders rSrLrLmsBPs###Us###Vs###&3+&& #FFFF  ( (   PV   #DV_n##    v e       &~  *) ( 87 : &~ ,+ v  K 21  ! "T# $.-% &43' ("\) rU)r)rHr*r1r"r%r3r4r2r+r,r-r.r/r0r5r9r:rv2str | list[str] | list[Path] | IO[str] | IO[bytes]dict[str, object] | None CredentialProviderBuilder | Nonect|tr|}d}ng}|r"t|}nd}tj||||||t ||| f||| | | |||| |||d }t |S)N) cloud_optionsr4r+r,r.r/r0r5r-r9r:rv)r\r]itemsrnew_from_parquetrr)r;r)rHr*r1r"r%r3r4r2r+r,r-r.r/r0r5r9r:rvrfpylfs rSr|r|=s.&$446677  ' ^-=>> &/%+1 -3!)   D, D>>rU).r;rr(r<r)r=r"r>r%r?r*rr+r@r,rAr-r@r.rBr/rBr0r@r1r@r2r@r3rCr4rDr5r?r6r@r7rCr8r@r9r>r:r@rEr)r;rVr(r<r3rCr7rCr8r@r1r@rEr)r;rkrErl)r;rkrErs)*r;rr)r=r"r>r%r?r*rr+r@r,rAr-r@r.rBr/rBr0r@r1r@r2r@rHr@r3rCr4rDr5r?r9r>r:r@rvrwrEr)*r;r~r)r=rHr@r*rr1r@r"r>r%r?r3rr4rr2r@r+r@r,rAr-r@r.rBr/rBr0r@r5r?r9r>r:r@rvrwrEr)< __future__r contextlibr`pathlibrtypingrrrpolars.functions functionsrNpolarsrrdpolars._utils.deprecationr polars._utils.unstabler polars._utils.variousr r r polars._utils.wraprpolars.convertrpolars.dependenciesrpolars.io._utilsrr,polars.io.cloud.credential_provider._builderrsuppress ImportError polars.polarsrrrurrqrrrrpolars._typingrrrpolars.io.cast_optionsrpolars.io.cloudrr rTrKrLr|rUrSrs"""""" ))))))))))%%%%%%AAAAAA999999 ('''''%%%%%%//////Z%%JJ))))))MMMMMMIIIIIIJJJJJJJJJJJJJJJ W5555555555GGGGGGGGGG666666::::::VVVVVV-/?RRR/1CXVVV-1!%!'%) $%)!%-1OU-1%)"'/UUUUUWVSRUD-1-1-13!3!3!3!3!3!l((((.****6-/?RRR/1CXVVV!%!'%) $%)!%-1OU%)"'+/+KKKKKWVSRKb!'!%04<@%) $%)!%%)"'+/+99999999s1BBB