q-PhHddlmZddlZddlZddlmZddlmZmZm Z m Z ddl m Z ddlmZddlmZddlmZmZmZddlmZmZddlmZdd lmZmZmZm Z m!Z!dd l"m#Z#ej$e%5dd l&m'Z'm(Z(dd l&m)Z*dddn #1swxYwYerdd l+m,Z,ddl-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4edddedddddddddddddEd*Z5ddddddd+dFd-Z6edddeddddddddddd.dGd/Z7dddddd0dHd1Z8dId3Z)edddedddddddddd4dd5dddddd6dJdDZ9dS)K) annotationsN)Path)IO TYPE_CHECKINGAnyLiteral)deprecate_renamed_parameter)is_path_or_str_sequenceis_str_sequencenormalize_filepath)wrap_dfwrap_ldf)import_optional)is_glob_pattern is_local_fileparse_columns_argparse_row_index_argsprepare_file_arg)!_init_credential_provider_builder) PyDataFrame PyLazyFrame)read_ipc_schema)Sequence) DataFrameDataType LazyFrame) SchemaDict)CredentialProviderFunctionrow_count_namerow_index_namez0.20.4)versionrow_count_offsetrow_index_offsetFT)columnsn_rows use_pyarrow memory_mapstorage_optionsr r#rechunksourcestr | Path | IO[bytes] | bytesr$list[int] | list[str] | Noner% int | Noner&boolr'r(dict[str, Any] | None str | Noneintr)returnrc t|x} ttfrt| dx} st jddkr| r|rd} t | t||||||} |rXt|dtr(| tj |} n| |} | } | S|r|r|sd} t | t|||5}|rtd d d }|||| }t j||} || ||} || d|} | cdddSt+|||||||cdddS#1swxYwYdS)a Read into a DataFrame from Arrow IPC (Feather v2) file. See "File or Random Access format" on https://arrow.apache.org/docs/python/ipc.html. Arrow IPC files are also known as Feather (v2) files. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path to a file or a file-like object (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). If `fsspec` is installed, it will be used to open remote files. For file-like objects, the stream position may not be updated accordingly after reading. columns Columns to select. Accepts a list of column indices (starting at zero) or a list of column names. n_rows Stop reading from IPC file after reading `n_rows`. Only valid when `use_pyarrow=False`. use_pyarrow Use pyarrow or the native Rust reader. memory_map Try to memory map the file. This can greatly improve performance on repeated queries as the OS may cache pages. Only uncompressed IPC files can be memory mapped. storage_options Extra options that make sense for `fsspec.open()` or a particular storage connection, e.g. host, port, username, password, etc. row_index_name Insert a row index column with the given name into the DataFrame as the first column. If set to `None` (default), no row index column is created. row_index_offset Start the row index at this offset. Cannot be negative. Only used if `row_index_name` is set. rechunk Make sure that all data is contiguous. Returns ------- DataFrame See Also -------- scan_ipc : Lazily read from an IPC file or multiple files via glob patterns. Warnings -------- Calling `read_ipc().lazy()` is an antipattern as this forces Polars to materialize a full csv file and therefore cannot push any optimizations into the reader. Therefore always prefer `scan_ipc` if you want to work with `LazyFrame` s. If `memory_map` is set, the bytes on disk are mapped 1:1 to memory. That means that you cannot write to the same filename. E.g. `pl.read_ipc("my_file.arrow").write_ipc("my_file.arrow")` will fail. zhf://POLARS_FORCE_ASYNC1z4`use_pyarrow=True` is not supported for Hugging Face)r%r(r r#r)rzF`n_rows` cannot be used with `use_pyarrow=True` and `memory_map=False`r&r(zpyarrow.featherz8is required when using 'read_ipc(..., use_pyarrow=True)' err_prefix err_suffix)r'r$r)Nr$r%r r#r)r') isinstancestrr startswithosgetenv ValueErrorscan_ipcr1selectFnthcollectrr read_tableplr _from_arrowwith_row_indexslice_read_ipc_impl)r*r$r%r&r'r(r r#r)vis_hfmsglfdfdatapyarrow_feathertbls W/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/polars/io/ipc/functions.pyread_ipcrW)sX ;1d ,,! a&&##G,, , ! 9) * *c 1 1  "[ "HCS// !  +)-      ('!*c** (YYquW~~..YYw'' ZZ\\ vjVoo K     -!UO ",,%-C ))#w)??B)&&~7GHH!XXa(('        * )-!   +                  sA?G *G  GGr<$Sequence[int] | Sequence[str] | Nonec rt|ttfrt|d}t|tr|g}t|trt |rt |rwt |||||}||}nJt|dr(| |}nd} t| |St|\} }tj ||| |t|||} t| S)NFcheck_not_directory)r%r)r r#) allow_strzccannot use glob patterns and integer based projection as `columns` argument Use columns: List[str])r')r=r>rr rrrCrGr rD TypeErrorrrrWrr ) r*r$r%r r#r)r'scanrRrP projectionpydfs rVrMrMsT&3+&&G#FFFF'3)&#?6#:#:}V?T?T )-     ?BB W 6 6 6 !W%%--//BB- C..  +G44J  ^-=>>    D 4==)r$r%r&r(r r#r)c t|||5}|rtddd} | |5} | } tj| |} || ||} || d|} | cdddcdddS#1swxYwYt|||||| cdddS#1swxYwYdS) a^ Read into a DataFrame from Arrow IPC record batch stream. See "Streaming format" on https://arrow.apache.org/docs/python/ipc.html. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path to a file or a file-like object (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). If `fsspec` is installed, it will be used to open remote files. For file-like objects, the stream position may not be updated accordingly after reading. columns Columns to select. Accepts a list of column indices (starting at zero) or a list of column names. n_rows Stop reading from IPC stream after reading `n_rows`. Only valid when `use_pyarrow=False`. use_pyarrow Use pyarrow or the native Rust reader. storage_options Extra options that make sense for `fsspec.open()` or a particular storage connection, e.g. host, port, username, password, etc. row_index_name Insert a row index column with the given name into the DataFrame as the first column. If set to `None` (default), no row index column is created. row_index_offset Start the row index at this offset. Cannot be negative. Only used if `row_index_name` is set. rechunk Make sure that all data is contiguous. Returns ------- DataFrame r6z pyarrow.ipcr7z?is required when using 'read_ipc_stream(..., use_pyarrow=True)'r8r;Nrr$r%r r#r)) rrRecordBatchStreamReaderread_allrIrrJrKrL_read_ipc_stream_impl) r*r$r%r&r(r r#r)rS pyarrow_ipcreaderrUrRs rVread_ipc_streamrisl K     )\K 44T:: foo''\--c7-CC!-**>;KLLB%!V,,B                       % )-    %                  s5*C)A'B=$ C)=C C)C C))C-0C-rcc t|ttfrt|d}t|tr|g}t |\}}t j||||t|||}t|S)NFrZ) r=r>rr rrrirr )r*r$r%r r#r)r_r`s rVrfrf;s&3+&&G#FFFF'3)+G44J  &^-=>>   D 4==radict[str, DataType]czt|ttfrt|d}t |S)a Get the schema of an IPC file without reading data. Parameters ---------- source Path to a file or a file-like object (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). For file-like objects, the stream position may not be updated accordingly after reading. Returns ------- dict Dictionary mapping column names to datatypes FrZ)r=r>rr _read_ipc_schema)r*s rVrrUs;"&3+&&G#FFFF F # ##raauto)r%cacher)r r#r(credential_providerr'retriesfile_cache_ttlhive_partitioning hive_schematry_parse_hive_datesinclude_file_pathsWstr | Path | IO[bytes] | bytes | list[str] | list[Path] | list[IO[bytes]] | list[bytes]rprq3CredentialProviderFunction | Literal['auto'] | Nonerrrsrt bool | NoneruSchemaDict | Nonervrwrcg}t|ttfrt|d}n5t|tr t |r d|D}n|}d}|}t |||d}~|r"t |}nd}tj |||||t||||| | | | | |}t|S)a Lazily read from an Arrow IPC (Feather v2) file or multiple files via glob patterns. This allows the query optimizer to push down predicates and projections to the scan level, thereby potentially reducing memory overhead. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path(s) to a file or directory When needing to authenticate for scanning cloud locations, see the `storage_options` parameter. n_rows Stop reading from IPC file after reading `n_rows`. cache Cache the result after reading. rechunk Reallocate to contiguous memory when all chunks/ files are parsed. row_index_name If not None, this will insert a row index column with give name into the DataFrame row_index_offset Offset to start the row index column (only use if the name is set) storage_options Options that indicate how to connect to a cloud provider. The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here: * `aws `_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. memory_map Try to memory map the file. This can greatly improve performance on repeated queries as the OS may cache pages. Only uncompressed IPC files can be memory mapped. retries Number of retries if accessing a cloud instance fails. file_cache_ttl Amount of time to keep downloaded cloud files since their last access time, in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable (which defaults to 1 hour) if not given. hive_partitioning Infer statistics and schema from Hive partitioned URL and use them to prune reads. This is unset by default (i.e. `None`), meaning it is automatically enabled when a single directory is passed, and otherwise disabled. hive_schema The column names and data types of the columns by which the data is partitioned. If set to `None` (default), the schema of the Hive partitions is inferred. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. try_parse_hive_dates Whether to try parsing hive values as date/datetime types. include_file_paths Include the path of the source file(s) as a column with this name. FrZc0g|]}t|dS)FrZ)r ).0r*s rV zscan_ipc..s5#6uEEEraN scan_parquet) cloud_optionsrqrrrsrtrurvrw) r=r>rr listr ritemsr new_from_ipcrr)r*r%rpr)r r#r(rqr'rrrsrtrurvrwsources_credential_provider_builderpylfs rVrCrCls-TGIG&3+&& #FFFF FD ! !  "6 * * $GG G A"CV_n## 446677  # ^-=>>%7%+1-   D D>>ra)r*r+r$r,r%r-r&r.r'r.r(r/r r0r#r1r)r.r2r)r*r+r$rXr%r-r r0r#r1r)r.r'r.r2r)r*r+r$r,r%r-r&r.r(r/r r0r#r1r)r.r2r)r*r+r$rXr%r-r r0r#r1r)r.r2r)r*r+r2rk) r*rxr%r-rpr.r)r.r r0r#r1r(r/rqryr'r.rrr1rsr-rtrzrur{rvr.rwr0r2r): __future__r contextlibr@pathlibrtypingrrrrpolars._reexport _reexportrIpolars.functions functionsrEpolars._utils.deprecationr polars._utils.variousr r r polars._utils.wrapr rpolars.dependenciesrpolars.io._utilsrrrrr,polars.io.cloud.credential_provider._builderrsuppress ImportError polars.polarsrrrrmcollections.abcrpolarsrrrpolars._typingrpolars.io.cloudrrWrMrirfrCrarVrsp"""""" 222222222222AAAAAA 10000000//////Z%%BB66666666AAAAAABBBBBBBBBBBBBBB;((((((5555555555))))))::::::-/?RRR/1CXVVV-1-1!%L L L L L WVSRL d59!%,,,,,,^-/?RRR/1CXVVV-1-1!%M M M M M WVSRM f59!%4$$$$.-/?RRR/1CXVVV!%-1OU!%%)%)!%%)3TTTTTWVSRTTTs/B  BB