q-Phl6ddlmZddlZddlmZddlmZmZddlm Z ddl m Z m Z m Z mZddlmZddlmZmZdd lmZmZdd lmZdd lmZdd lmZeje5dd lm Z m!Z!dddn #1swxYwYe rddlm"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)ddedddddddddddddd9d1Z*ed2d#d34ed5d%d34ddedddddddddddddd:d8Z+dS);) annotationsN)Sequence)BytesIOStringIO)Path)IO TYPE_CHECKINGAnyLiteral)deprecate_renamed_parameter)is_path_or_str_sequencenormalize_filepath)wrap_dfwrap_ldf)N_INFER_DEFAULT)parse_row_index_args)!_init_credential_provider_builder) PyDataFrame PyLazyFrame)IOBase) DataFrame LazyFrame)SchemaDefinition)CredentialProviderFunctioniFauto)schemaschema_overridesinfer_schema_length batch_sizen_rows low_memoryrechunkrow_index_namerow_index_offset ignore_errorsstorage_optionscredential_providerretriesfile_cache_ttlinclude_file_pathssource4str | Path | list[str] | list[Path] | IOBase | bytesrSchemaDefinition | Nonerr int | Noner r!r"boolr#r$ str | Noner%intr&r'dict[str, Any] | Noner(3CredentialProviderFunction | Literal['auto'] | Noner)r*r+returnrc<t|ttfst|tr$|r"t|dttfst|tr3t |}tj || ||}t|}|r| |}|St| || d}~ t|f||||||||| | || | ||dS)u~ Read into a DataFrame from a newline delimited JSON file. Parameters ---------- source Path to a file or a file-like object (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). For file-like objects, the stream position may not be updated accordingly after reading. schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict The DataFrame schema may be declared in several ways: * As a dict of {name:type} pairs; if type is None, it will be auto-inferred. * As a list of column names; in this case types are automatically inferred. * As a list of (name,type) pairs; this is equivalent to the dictionary form. If you supply a list of column names that does not match the names in the underlying data, the names given here will overwrite them. The number of names given in the schema should match the underlying data dimensions. schema_overrides : dict, default None Support type specification or override of one or more columns; note that any dtypes inferred from the schema param will be overridden. infer_schema_length The maximum number of rows to scan for schema inference. If set to `None`, the full data may be scanned *(this is slow)*. batch_size Number of rows to read in each batch. n_rows Stop reading from JSON file after reading `n_rows`. low_memory Reduce memory pressure at the expense of performance. rechunk Reallocate to contiguous memory when all chunks/ files are parsed. row_index_name If not None, this will insert a row index column with give name into the DataFrame row_index_offset Offset to start the row index column (only use if the name is set) ignore_errors Return `Null` if parsing fails because of schema mismatches. storage_options Options that indicate how to connect to a cloud provider. The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here: * `aws `_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \ `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. retries Number of retries if accessing a cloud instance fails. file_cache_ttl Amount of time to keep downloaded cloud files since their last access time, in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable (which defaults to 1 hour) if not given. include_file_paths Include the path of the source file(s) as a column with this name. See Also -------- scan_ndjson : Lazily read from an NDJSON file or multiple files via glob patterns. Warnings -------- Calling `read_ndjson().lazy()` is an antipattern as this forces Polars to materialize a full ndjson file and therefore cannot push any optimizations into the reader. Therefore always prefer `scan_ndjson` if you want to work with `LazyFrame` s. Examples -------- >>> from io import StringIO >>> json_str = '{"foo":1,"bar":6}\n{"foo":2,"bar":7}\n{"foo":3,"bar":8}\n' >>> pl.read_ndjson(StringIO(json_str)) shape: (3, 2) ┌─────┬─────┐ │ foo ┆ bar │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪═════╡ │ 1 ┆ 6 │ │ 2 ┆ 7 │ │ 3 ┆ 8 │ └─────┴─────┘ r)r&rr read_ndjson)rrrr r!r"r#r$r%r&r+r)r'r(r*) isinstancestrrrrrgetvalueencoderr7rheadr scan_ndjsoncollect)r,rrrr r!r"r#r$r%r&r'r(r)r*r+pydfdfcredential_provider_builders P/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/polars/io/ndjson.pyr7r7sMn 6C;'' vx ( (  6!9sDk22  fh ' ' 9V__..557788F& '-    T]]  !B "CV_m##   )/%)#-'7%!   "gii#row_count_namez0.20.4)versionrow_count_offsetcstr | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]]rcg}t|ttfrt|d}n5t|tr t |r d|D}n|}d}|dkrd}t |t| || d}~ | r"t | } nd} tj |||||||||t|| | || | ||}t|S) aM Lazily read from a newline delimited JSON file or multiple files via glob patterns. This allows the query optimizer to push down predicates and projections to the scan level, thereby potentially reducing memory overhead. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path to a file. schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict The DataFrame schema may be declared in several ways: * As a dict of {name:type} pairs; if type is None, it will be auto-inferred. * As a list of column names; in this case types are automatically inferred. * As a list of (name,type) pairs; this is equivalent to the dictionary form. If you supply a list of column names that does not match the names in the underlying data, the names given here will overwrite them. The number of names given in the schema should match the underlying data dimensions. schema_overrides : dict, default None Support type specification or override of one or more columns; note that any dtypes inferred from the schema param will be overridden. infer_schema_length The maximum number of rows to scan for schema inference. If set to `None`, the full data may be scanned *(this is slow)*. batch_size Number of rows to read in each batch. n_rows Stop reading from JSON file after reading `n_rows`. low_memory Reduce memory pressure at the expense of performance. rechunk Reallocate to contiguous memory when all chunks/ files are parsed. row_index_name If not None, this will insert a row index column with give name into the DataFrame row_index_offset Offset to start the row index column (only use if the name is set) ignore_errors Return `Null` if parsing fails because of schema mismatches. storage_options Options that indicate how to connect to a cloud provider. The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here: * `aws `_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. retries Number of retries if accessing a cloud instance fails. file_cache_ttl Amount of time to keep downloaded cloud files since their last access time, in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable (which defaults to 1 hour) if not given. include_file_paths Include the path of the source file(s) as a column with this name. Fcheck_not_directoryc0g|]}t|dS)FrI)r).0r,s rB zscan_ndjson..7s5#6uEEErCNrz('infer_schema_length' should be positiver=)rrrr r!r"r# row_indexr&r+r) cloud_optionsr(r*) r8r9rrlistr ValueErrorritemsrnew_from_ndjsonrr)r,rrrr r!r"r#r$r%r&r'r(r)r*r+sourcesmsgrApylfs rBr=r=sHVIKG&3+&& #FFFF FD ! !  "6 * * $GG Ga8oo"CV_m## 446677  &/)&~7GHH#-%7%!   D$ D>>rC)"r,r-rr.rr.rr/r r/r!r/r"r0r#r0r$r1r%r2r&r0r'r3r(r4r)r2r*r/r+r1r5r)"r,rGrr.rr.rr/r r/r!r/r"r0r#r0r$r1r%r2r&r0r'r3r(r4r)r2r*r/r+r1r5r), __future__r contextlibcollections.abcriorrpathlibrtypingrr r r polars._utils.deprecationr polars._utils.variousr rpolars._utils.wraprrpolars.datatypesrpolars.io._utilsr,polars.io.cloud.credential_provider._builderrsuppress ImportError polars.polarsrrrpolarsrrpolars._typingrpolars.io.cloudrr7r=rCrBrjs""""""$$$$$$ 222222222222AAAAAAMMMMMMMM00000000,,,,,,111111Z%%7766666666777777777777777;++++++++//////:::::: '+04&5!!%-1OU!%%)#ggggggT-/?RRR/1CXVVV'+04&5!!%-1OU!%%)7YYYYYWVSRYYYs# A88A<?A<