q-Ph&ddlmZddlZddlZddlmZddlmZmZddl m Z ddl m Z m Z mZmZmZddlmZddlmZddlmZddlmZmZmZmZmZdd lm Z m!Z!dd l"m#Z#m$Z$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*dd l+m,Z,dd l-m.Z.m/Z/ddl0m1Z1ej2e35ddl4m5Z5m6Z6dddn #1swxYwYe r$ddlm7Z7ddl8m9Z9m:Z:ddl;mZ>ddl?m@Z@ddl+mAZAedddedddeddddddddd dddddd!d!d!dde#d"dd#d!d!d!ddddd$d%dd!d!dd&!dtdUZBddddd dddddd!d!d!de#d"dd#d!d!dddd$d%dd!d!ddVdud[ZCedddedddeddddddddd dddd!d!d!de#d\dd#d!d!dddd$d%dd!d!d]dvd`ZDedddedddeddddddd dddddd!d!ddde#dd#d!d!dddd!d%ddd!d!dddadbdddc"dwdoZEdddd dddddd!d!dde#dd#d!d!dddd!d%ddd!ddddbdddp dxdsZFdS)y) annotationsN)Sequence)BytesIOStringIO)Path)IO TYPE_CHECKINGAnyCallableLiteral)deprecate_renamed_parameter)_process_null_valuesis_path_or_str_sequenceis_str_sequencenormalize_filepathqualified_type_name)wrap_dfwrap_ldf)N_INFER_DEFAULTStringparse_into_dtype)is_glob_patternparse_columns_argparse_row_index_argsprepare_file_arg)!_init_credential_provider_builder)_check_arg_is_1byte_update_columns)BatchedCsvReader) PyDataFrame PyLazyFrame)Mapping) DataFrame LazyFrame) CsvEncodingPolarsDataType SchemaDict)CredentialProviderFunction)CredentialProviderBuilderdtypesschema_overridesz0.20.31)versionrow_count_namerow_index_namez0.20.4row_count_offsetrow_index_offsetT,"Fi utf8i )! has_headercolumns new_columns separatorcomment_prefix quote_char skip_rows skip_linesschemar+ null_valuesmissing_utf8_is_empty_string ignore_errorstry_parse_dates n_threads infer_schemainfer_schema_length batch_sizen_rowsencoding low_memoryrechunk use_pyarrowstorage_optionsskip_rows_after_headerr.r0 sample_sizeeol_charraise_if_emptytruncate_ragged_lines decimal_commaglobsource(str | Path | IO[str] | IO[bytes] | bytesr5boolr6$Sequence[int] | Sequence[str] | Noner7Sequence[str] | Noner8strr9 str | Noner:r;intr<r=SchemaDict | None>Mapping[str, PolarsDataType] | Sequence[PolarsDataType] | Noner>+str | Sequence[str] | dict[str, str] | Noner?r@rArB int | NonerCrDrErFrGCsvEncoding | strrHrIrJrKdict[str, Any] | NonerLrMrNrOrPrQrRreturnr#c!  6td|dtd|dtd|dt|\}"}|pi}|r-|s+|D](}#|#dsd}$t|$) -t t t fsd }$t|$|r |||s| }d }%|r|s d |D}%n|}%|s|"r d |"D}%t|d d|| 5}&dd l }'dd l }( |'j |&|'j ||| ||'j ||r|nd|d uo|dk|'j d |%| })nJ#|'j$r=}*|sdt#|*vrt%jcYd }*~*cd d d Sd }*~*wwxYw d d d n #1swxYwY|s$|)d|)jD})t$j|)|}+|rt/|+|S|+S|"r rt t0rt3|"t3 krd}$t|$t4gt7|"dzz},t9|"D]#\}-}.|-t3 kr |-|,|.<$|, |re rct t0rNt3|t3 krd}$t|$t t;|  |r rt t rud }/|rIt3|t3|krd}$t|$|dt3|}/n|sm|"r@|r1t3|t3|krd}$t|$d|"D}/ndt=dt3|dzD}/net3 t3|krE fd|dt3 D}0t3|0t3 kr|0 |/rRt t r=t t;||/66fd D |sd}t t }1|dv}2tAj!dd kptAj!d!d k}3|3s`t |x}4t"tDfrdt#|4d"stAj!d#d kr)|1s&|2r#t |t"tDfrtG|d$}ntI|d%r d&|D}|3s*|1rd'}$t|$|2sd(|d)}$t|$tK|fid*|d|d+|d|d,|d-|d. d/| d0| d1| d2| d3|d4|d5|d6|d7|d8|d9|d:|d;|d|d<|d=|d>| d?|!}5|r|5&|}5n)|"r'|5&tOj(|"}5|5)}+nt||d|| 5}&tU|&fid*|d@|r|n|"d|d+|d|d,|d-|d. d/| d0| d1| d2| d3|dA|d4|dB|d5|d6|dCkr|ndDd7|d8|d9|d:|d;|d|d<|d=|d>| d?|!}+d d d n #1swxYwY|rt/|+|S|+S)Eu" Read a CSV file into a DataFrame. .. versionchanged:: 0.20.31 The `dtypes` parameter was renamed `schema_overrides`. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path to a file or a file-like object (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). If `fsspec` is installed, it will be used to open remote files. For file-like objects, the stream position may not be updated accordingly after reading. has_header Indicate if the first row of the dataset is a header or not. If set to False, column names will be autogenerated in the following format: `column_x`, with `x` being an enumeration over every column in the dataset, starting at 1. columns Columns to select. Accepts a list of column indices (starting at zero) or a list of column names. new_columns Rename columns right after parsing the CSV file. If the given list is shorter than the width of the DataFrame the remaining columns will have their original name. separator Single byte character to use as separator in the file. comment_prefix A string used to indicate the start of a comment line. Comment lines are skipped during parsing. Common examples of comment prefixes are `#` and `//`. quote_char Single byte character used for csv quoting, default = `"`. Set to None to turn off special handling and escaping of quotes. skip_rows Start reading after ``skip_rows`` rows. The header will be parsed at this offset. Note that we respect CSV escaping/comments when skipping rows. If you want to skip by newline char only, use `skip_lines`. skip_lines Start reading after `skip_lines` lines. The header will be parsed at this offset. Note that CSV escaping will not be respected when skipping lines. If you want to skip valid CSV rows, use ``skip_rows``. schema Provide the schema. This means that polars doesn't do schema inference. This argument expects the complete schema, whereas `schema_overrides` can be used to partially overwrite a schema. Note that the order of the columns in the provided `schema` must match the order of the columns in the CSV being read. schema_overrides Overwrite dtypes for specific or all columns during schema inference. null_values Values to interpret as null values. You can provide a: - `str`: All values equal to this string will be null. - `List[str]`: All values equal to any string in this list will be null. - `Dict[str, str]`: A dictionary that maps column name to a null value string. missing_utf8_is_empty_string By default a missing value is considered to be null; if you would prefer missing utf8 values to be treated as the empty string you can set this param True. ignore_errors Try to keep reading lines if some lines yield errors. Before using this option, try to increase the number of lines used for schema inference with e.g `infer_schema_length=10000` or override automatic dtype inference for specific columns with the `schema_overrides` option or use `infer_schema=False` to read all columns as `pl.String` to check which values might cause an issue. try_parse_dates Try to automatically parse dates. Most ISO8601-like formats can be inferred, as well as a handful of others. If this does not succeed, the column remains of data type `pl.String`. If `use_pyarrow=True`, dates will always be parsed. n_threads Number of threads to use in csv parsing. Defaults to the number of physical cpu's of your system. infer_schema When `True`, the schema is inferred from the data using the first `infer_schema_length` rows. When `False`, the schema is not inferred and will be `pl.String` if not specified in `schema` or `schema_overrides`. infer_schema_length The maximum number of rows to scan for schema inference. If set to `None`, the full data may be scanned *(this is slow)*. Set `infer_schema=False` to read all columns as `pl.String`. batch_size Number of lines to read into the buffer at once. Modify this to change performance. n_rows Stop reading from CSV file after reading `n_rows`. During multi-threaded parsing, an upper bound of `n_rows` rows cannot be guaranteed. encoding : {'utf8', 'utf8-lossy', 'windows-1252', 'windows-1252-lossy', ...} Lossy means that invalid utf8 values are replaced with `�` characters. When using other encodings than `utf8` or `utf8-lossy`, the input is first decoded in memory with python. Defaults to `utf8`. low_memory Reduce memory pressure at the expense of performance. rechunk Make sure that all columns are contiguous in memory by aggregating the chunks into a single array. use_pyarrow Try to use pyarrow's native CSV parser. This will always parse dates, even if `try_parse_dates=False`. This is not always possible. The set of arguments given to this function determines if it is possible to use pyarrow's native parser. Note that pyarrow and polars may have a different strategy regarding type inference. storage_options Extra options that make sense for `fsspec.open()` or a particular storage connection. e.g. host, port, username, password, etc. skip_rows_after_header Skip this number of rows when the header is parsed. row_index_name Insert a row index column with the given name into the DataFrame as the first column. If set to `None` (default), no row index column is created. row_index_offset Start the row index at this offset. Cannot be negative. Only used if `row_index_name` is set. sample_size Set the sample size. This is used to sample statistics to estimate the allocation needed. .. deprecated:: 1.10.0 This parameter is now a no-op. eol_char Single byte end of line character (default: `\n`). When encountering a file with windows line endings (`\r\n`), one can go with the default `\n`. The extra `\r` will be removed when processed. raise_if_empty When there is no data in the source, `NoDataError` is raised. If this parameter is set to False, an empty DataFrame (with no columns) is returned instead. truncate_ragged_lines Truncate lines that are longer than the schema. decimal_comma Parse floats using a comma as the decimal separator instead of a period. glob Expand path given via globbing rules. Returns ------- DataFrame See Also -------- scan_csv : Lazily read from a CSV file or multiple files via glob patterns. Warnings -------- Calling `read_csv().lazy()` is an antipattern as this forces Polars to materialize a full csv file and therefore cannot push any optimizations into the reader. Therefore always prefer `scan_csv` if you want to work with `LazyFrame` s. Notes ----- If the schema is inferred incorrectly (e.g. as `pl.Int64` instead of `pl.Float64`), try to increase the number of lines used to infer the schema with `infer_schema_length` or override the inferred dtype for those columns with `schema_overrides`. Examples -------- >>> pl.read_csv("data.csv", separator="|") # doctest: +SKIP Demonstrate use against a BytesIO object, parsing string dates. >>> from io import BytesIO >>> data = BytesIO( ... b"ID,Name,Birthday\n" ... b"1,Alice,1995-07-12\n" ... b"2,Bob,1990-09-20\n" ... b"3,Charlie,2002-03-08\n" ... ) >>> pl.read_csv(data, try_parse_dates=True) shape: (3, 3) ┌─────┬─────────┬────────────┐ │ ID ┆ Name ┆ Birthday │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ date │ ╞═════╪═════════╪════════════╡ │ 1 ┆ Alice ┆ 1995-07-12 │ │ 2 ┆ Bob ┆ 1990-09-20 │ │ 3 ┆ Charlie ┆ 2002-03-08 │ └─────┴─────────┴────────────┘ r8F can_be_emptyr:TrNcolumn_aspecified column names do not start with 'column_', but autogenerated header names were requestedN1`schema_overrides` should be of type list or dictcHg|]}dt|dddz  S)fNrZ.0columns W/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/polars/io/csv/functions.py zread_csv..7s4"S"S"S#.>s!MMMJ/://MMMrr)rGrJrOrKr)r;skip_rows_after_namesautogenerate_column_namesrGr2) delimiterr: double_quote) column_typesinclude_columnsinclude_missing_columnsz Empty CSVcHg|]}dt|dddz S)rerkNrlrms rprqzread_csv..gs4PPPV03vabbz??Q.00PPPrr)rICmore schema overrides are specified than there are selected columnsrkzCmore dtypes overrides are specified than there are selected columnsCmore new column names are specified than there are selected columnscg|] }d|dz Srerkrtrus rprqzread_csv..2###3=.j1n..###rrcg|]}d|Srertrus rprqzread_csv..0###"+j**###rrc(g|]}|v|Srtrtrnnew_column_namer+s rprqzread_csv..4'&*:::%_5:::rrcDi|]\}}|||Srtgetrn column_name column_dtypenew_to_currents rp zread_csv..?   -K""; <r3 utf8-lossyPOLARS_FORCE_NEW_STREAMING1POLARS_AUTO_NEW_STREAMINGzhf://POLARS_FORCE_ASYNCcheck_not_directory allow_strc0g|]}t|dSFrrrnrSs rprqzread_csv..s5#6uEEErrzCpassing a list to `schema_overrides` is unsupported for hf:// pathszunsupported encoding z for hf:// pathsr5r9r;r<r+r=r>r?r@rArDrFrGrHrIrLr.r0rOrPrQrRr6rBrErr3)+rr startswith ValueError isinstancedictr TypeErrorrpyarrow pyarrow.csvcsvread_csv ReadOptions ParseOptionsConvertOptions ArrowInvalidrXplr#rename_columns column_names _from_arrowrlistlenrmax enumerateziprangeitemsosgetenvrrr_scan_csv_implselectFnthcollect_read_csv_impl)7rSr5r6r7r8r9r:r;r<r=r+r>r?r@rArBrCrDrErFrGrHrIrJrKrLr.r0rMrNrOrPrQrR projectionromsgr|datapartblerrdf dtypes_listidxrvcurrent_columns dtype_listschema_overrides_is_listencoding_supported_in_lazy new_streamingvlfrs7 ` @rprr.s L YUCCCC jtDDDD H5AAAA+G44J%+O&z& & &F$$Y// &E!oo%  &#J4*--#Bnn C  $ N     04  * *#T"S7"S"S"S") N: NNM*MMMO  )+    ! &      &fooF&&"+.D6@.!) ' F''"+1;#F::%/t%;%Q c@Q( F))%)(70=*&? & & &![C%@%@|~~%%%%%C! &! &! &! &! &! &! &! &< &'! &! &! &! &! &! &! &! &! &! &! &! &! &! &! &F $$PPs?OPPPC\ % %c7 % ; ;  4"2{33 3  '& ':6F+M+M ' z??S!122 2 2WCS// !.4HJ!8K,L (44 @ @OCS)*****:3*? J'&@#@ 3CT(J(J@ w<<#.// / /WCS// !  G-= > >??='=J7G,N,N= / 27||c+....!oo%&a#k*:*:&:;OO% 2 *s7||c+.>.>>>#%S//)##AK### ##&+As;/?/?!/C&D&D####$$K(8(888 +6q3?O;P;P7P+Q z??c*:&;&;;;'1$  z*:DAA !#k?"C"CDDN    1A1G1G1I1I       **:HEE!)-C!C .//36 9 90 1 1S 8 m;1d ,,m FF  g & & m .//36607.7 fsDk * * 'EJJJFF $Vu = = = $F  &' &[ oo%- &HhHHH oo%     !z  i *>  "z   i  "z .- 6 $  *F)E (- ,O !4 3 6 X! ""z# $G% &$:#9' (*>) *.-+ ,X- .*>/ 0#8"71 2(-3 45 :  .7##BB  .15,,--B ZZ\\ )+    %  %:$+: $)  .~  &: $)&:"2!1v(K.J-I,m!0$) %8$7!"&:#$v%&&.%=%=6'(&:)* +,(>'=-. .~/0"2!112"34 .~56'<&;78,m9:T;B% % % % % % % % % % % % % % % N0r;/// IsV8 GAG? G &G/G0GGGGG4A,\,,\03\0)r5r6r8r9r:r;r<r=r+r>r?r@rArBrDrErFrGrHrIrLr.r0rMrNrOrPrQrRstr | Path | IO[bytes] | bytesNone | SchemaDict.None | (SchemaDict | Sequence[PolarsDataType])r%cRt|ttfrt|d}nfd}t|tr|}t|t r&|}d}d} | t| trAg}| D])\}!}"| |!t|"f*n9t| tr| } n!dt| }#t|#t| }$t|tr|g}t|trt!|rd}%|t|}%| d}#t#|#ddlm}&|&|fid|d|d |d |d |d |d |d|%d| d| d| d|d|d|d|d|d|d|d|d|d|d|d|}'||'St+|dr'|'|Sd}#t#|#t/|\}(}t1j||||| ||||(|||||||| ||||$| | |t5||f|||||d })t7|)S)!NFrz7`schema_overrides` should be of type list or dict, got zcannot use glob patterns and unnamed dtypes as `schema_overrides` argument Use `schema_overrides`: Mapping[str, Type[DataType]]r)scan_csvr5r8r9r:r;r<r=r+r>r?r@rDrFrHrIrLr.r0rNrOrPrQrRrzccannot use glob patterns and integer based projection as `columns` argument Use columns: List[str])rNrOrPrQr=)rrXrrrgetvaluerencoderrappendrrrrrrrpolarsrrrrrr rrr)*rSr5r6r8r9r:r;r<r=r+r>r?r@rArBrDrErFrGrHrIrLr.r0rMrNrOrPrQrRpathr dtype_slicekrrprocessed_null_values dtypes_dictrscanrpydfs* rprrGsD&3+&&0!&eDDD fg & & '__&&F fh ' ' 0__&&--//F>BJ37K# & - - !J(..00 < <1!!1&6q&9&9":;;;; < (( 3 3 !*KKuL_`pLqLquuCC.. 0=='3)&#/"?6#:#:/"  !z**K  "K S// !######x    !z  i *>  "z   i  "z 6 )[ $  *F)E (- !4 3 6 "z G! "$:#9# $*>% &.-' (X) **>+ ,#8"7- .(-/ 01 4 ?<<>> ! W 6 6 6 ";;w''//11 1- S// !+G44J   $^-=>>3 4%3#=   D@ 4==rriP)r5r6r7r8r9r:r;r+r>r?r@rArBrDrErFrGrHrIrLr.r0rMrNrOrPrQ str | Pathrcv$t|\}}|r-|s+|D](}|dsd}t|)|rrttrt |t krd}t|t gt|dzz}t|D]#\} }!| t kr | ||!<$||rercttrNt |t krd}t|tt||rrttrud}"|rIt |t |krd}t||dt |}"n|sm|r@|r1t |t |krd}t|d|D}"nd tdt |dzD}"net t |krEfd |dt D}#t |#t kr|#|"rRttr=tt||"$$fd  Dt|fid |d |r|n|d|d|d|d|dd| d| d| d| d| d|d|d|d|dkr|ndd|d|d |d!|d"|d#|d$|d%|d&|d'|S)(uF Read a CSV file in batches. Upon creation of the `BatchedCsvReader`, Polars will gather statistics and determine the file chunks. After that, work will only be done if `next_batches` is called, which will return a list of `n` frames of the given batch size. .. versionchanged:: 0.20.31 The `dtypes` parameter was renamed `schema_overrides`. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path to a file or a file-like object (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). If `fsspec` is installed, it will be used to open remote files. For file-like objects, the stream position may not be updated accordingly after reading. has_header Indicate if the first row of the dataset is a header or not. If set to False, column names will be autogenerated in the following format: `column_x`, with `x` being an enumeration over every column in the dataset, starting at 1. columns Columns to select. Accepts a list of column indices (starting at zero) or a list of column names. new_columns Rename columns right after parsing the CSV file. If the given list is shorter than the width of the DataFrame the remaining columns will have their original name. separator Single byte character to use as separator in the file. comment_prefix A string used to indicate the start of a comment line. Comment lines are skipped during parsing. Common examples of comment prefixes are `#` and `//`. quote_char Single byte character used for csv quoting, default = `"`. Set to None to turn off special handling and escaping of quotes. skip_rows Start reading after `skip_rows` lines. schema_overrides Overwrite dtypes during inference. null_values Values to interpret as null values. You can provide a: - `str`: All values equal to this string will be null. - `List[str]`: All values equal to any string in this list will be null. - `Dict[str, str]`: A dictionary that maps column name to a null value string. missing_utf8_is_empty_string By default a missing value is considered to be null; if you would prefer missing utf8 values to be treated as the empty string you can set this param True. ignore_errors Try to keep reading lines if some lines yield errors. First try `infer_schema_length=0` to read all columns as `pl.String` to check which values might cause an issue. try_parse_dates Try to automatically parse dates. Most ISO8601-like formats can be inferred, as well as a handful of others. If this does not succeed, the column remains of data type `pl.String`. n_threads Number of threads to use in csv parsing. Defaults to the number of physical cpu's of your system. infer_schema_length The maximum number of rows to scan for schema inference. If set to `0`, all columns will be read as `pl.String`. If set to `None`, the full data may be scanned *(this is slow)*. batch_size Number of lines to read into the buffer at once. Modify this to change performance. n_rows Stop reading from CSV file after reading `n_rows`. During multi-threaded parsing, an upper bound of `n_rows` rows cannot be guaranteed. encoding : {'utf8', 'utf8-lossy', ...} Lossy means that invalid utf8 values are replaced with `�` characters. When using other encodings than `utf8` or `utf8-lossy`, the input is first decoded in memory with python. Defaults to `utf8`. low_memory Reduce memory pressure at the expense of performance. rechunk Make sure that all columns are contiguous in memory by aggregating the chunks into a single array. skip_rows_after_header Skip this number of rows when the header is parsed. row_index_name Insert a row index column with the given name into the DataFrame as the first column. If set to `None` (default), no row index column is created. row_index_offset Start the row index at this offset. Cannot be negative. Only used if `row_index_name` is set. sample_size Set the sample size. This is used to sample statistics to estimate the allocation needed. .. deprecated:: 1.10.0 Is a no-op. eol_char Single byte end of line character (default: `\n`). When encountering a file with windows line endings (`\r\n`), one can go with the default `\n`. The extra `\r` will be removed when processed. raise_if_empty When there is no data in the source,`NoDataError` is raised. If this parameter is set to False, `None` will be returned from `next_batches(n)` instead. truncate_ragged_lines Truncate lines that are longer than the schema. decimal_comma Parse floats using a comma as the decimal separator instead of a period. Returns ------- BatchedCsvReader See Also -------- scan_csv : Lazily read from a CSV file or multiple files via glob patterns. Examples -------- >>> reader = pl.read_csv_batched( ... "./pdsh/tables_scale_100/lineitem.tbl", ... separator="|", ... try_parse_dates=True, ... ) # doctest: +SKIP >>> batches = reader.next_batches(5) # doctest: +SKIP >>> for df in batches: # doctest: +SKIP ... print(df) Read big CSV file in batches and write a CSV file for each "group" of interest. >>> seen_groups = set() >>> reader = pl.read_csv_batched("big_file.csv") # doctest: +SKIP >>> batches = reader.next_batches(100) # doctest: +SKIP >>> while batches: # doctest: +SKIP ... df_current_batches = pl.concat(batches) ... partition_dfs = df_current_batches.partition_by("group", as_dict=True) ... ... for group, df in partition_dfs.items(): ... if group in seen_groups: ... with open(f"./data/{group}.csv", "a") as fh: ... fh.write(df.write_csv(file=None, include_header=False)) ... else: ... df.write_csv(file=f"./data/{group}.csv", include_header=True) ... seen_groups.add(group) ... ... batches = reader.next_batches(100) rerfrrkNrrcg|] }d|dz Srrtrus rprqz$read_csv_batched..rrrcg|]}d|Srrtrus rprqz$read_csv_batched..rrrc(g|]}|v|Srtrtrs rprqz$read_csv_batched..rrrcDi|]\}}|||Srtrrs rprz$read_csv_batched..rrrr5r6r8r9r:r;r+r>r?r@rArBrDrErFrGrr3rHrIrLr.r0rNr7rOrPrQ)rrrrrrrrrrrrrr)%rSr5r6r7r8r9r:r;r+r>r?r@rArBrDrErFrGrHrIrLr.r0rMrNrOrPrQrrorrrrvrrrs% ` @rpread_csv_batchedrsz,G44J&z& & &F$$Y// &E!oo%  & '& ':6F+M+M ' z??S!122 2 2WCS// !.4HJ!8K,L (44 @ @OCS)*****:3*? J'&@#@ 3CT(J(J@ w<<#.// / /WCS// !  G-= > >??7'7J7G,N,N7 ) 27||c+....[ oo%&a#k*:*:&:;OO" 2 *s7||c+.>.>>>_C$S//)##AK### ##&+As;/?/?!/C&D&D####$$K(8(888 +6q3?O;P;P7P+Q z??c*:&;&;;;'1$  z*:DAA !#k?"C"CDDN    1A1G1G1I1I        : #2  )   &~   :  ) *)  K &B%A $m ( ) 0/ :  v! "&556# $:% &' ( 65) *&~+ ,*)- ./ 0 K1 2&~3 4435 6$m7 rrauto)"r5r8r9r:r;r<r=r+r>r?r@cachewith_column_namesrCrDrFrGrHrIrLr.r0rArNr7rOrPrQrRrKcredential_providerretriesfile_cache_ttlinclude_file_pathsqstr | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]] | list[bytes],SchemaDict | Sequence[PolarsDataType] | Nonerr'Callable[[list[str]], list[str]] | Noner3CredentialProviderFunction | Literal['auto'] | Nonerrrr$c"R|-t|ttfsd}#t|#s6t|tr!dt |}#t|#rM| rd}#t |#|r2t|trtt |}d1fd } td |d td |d t|ttfrt|d }nt|d r d|D}|sd}t|||d}$~t|fid|d |d|d |d|d|d|d|d| d| d| d| d| d|d|d |d!|d"|d#|d$|d%|d&|d'|d(|d)|d*|d+|d,| d-|d.|$d/|!d0|"S)2u$ Lazily read from a CSV file or multiple files via glob patterns. This allows the query optimizer to push down predicates and projections to the scan level, thereby potentially reducing memory overhead. .. versionchanged:: 0.20.31 The `dtypes` parameter was renamed `schema_overrides`. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path(s) to a file or directory When needing to authenticate for scanning cloud locations, see the `storage_options` parameter. has_header Indicate if the first row of the dataset is a header or not. If set to False, column names will be autogenerated in the following format: `column_x`, with `x` being an enumeration over every column in the dataset, starting at 1. separator Single byte character to use as separator in the file. comment_prefix A string used to indicate the start of a comment line. Comment lines are skipped during parsing. Common examples of comment prefixes are `#` and `//`. quote_char Single byte character used for csv quoting, default = `"`. Set to None to turn off special handling and escaping of quotes. skip_rows Start reading after ``skip_rows`` rows. The header will be parsed at this offset. Note that we respect CSV escaping/comments when skipping rows. If you want to skip by newline char only, use `skip_lines`. skip_lines Start reading after `skip_lines` lines. The header will be parsed at this offset. Note that CSV escaping will not be respected when skipping lines. If you want to skip valid CSV rows, use ``skip_rows``. schema Provide the schema. This means that polars doesn't do schema inference. This argument expects the complete schema, whereas `schema_overrides` can be used to partially overwrite a schema. Note that the order of the columns in the provided `schema` must match the order of the columns in the CSV being read. schema_overrides Overwrite dtypes during inference; should be a {colname:dtype,} dict or, if providing a list of strings to `new_columns`, a list of dtypes of the same length. null_values Values to interpret as null values. You can provide a: - `str`: All values equal to this string will be null. - `List[str]`: All values equal to any string in this list will be null. - `Dict[str, str]`: A dictionary that maps column name to a null value string. missing_utf8_is_empty_string By default a missing value is considered to be null; if you would prefer missing utf8 values to be treated as the empty string you can set this param True. ignore_errors Try to keep reading lines if some lines yield errors. First try `infer_schema=False` to read all columns as `pl.String` to check which values might cause an issue. cache Cache the result after reading. with_column_names Apply a function over the column names just in time (when they are determined); this function will receive (and should return) a list of column names. infer_schema When `True`, the schema is inferred from the data using the first `infer_schema_length` rows. When `False`, the schema is not inferred and will be `pl.String` if not specified in `schema` or `schema_overrides`. infer_schema_length The maximum number of rows to scan for schema inference. If set to `None`, the full data may be scanned *(this is slow)*. Set `infer_schema=False` to read all columns as `pl.String`. n_rows Stop reading from CSV file after reading `n_rows`. encoding : {'utf8', 'utf8-lossy'} Lossy means that invalid utf8 values are replaced with `�` characters. Defaults to "utf8". low_memory Reduce memory pressure at the expense of performance. rechunk Reallocate to contiguous memory when all chunks/ files are parsed. skip_rows_after_header Skip this number of rows when the header is parsed. row_index_name If not None, this will insert a row index column with the given name into the DataFrame. row_index_offset Offset to start the row index column (only used if the name is set). try_parse_dates Try to automatically parse dates. Most ISO8601-like formats can be inferred, as well as a handful of others. If this does not succeed, the column remains of data type `pl.String`. eol_char Single byte end of line character (default: `\n`). When encountering a file with windows line endings (`\r\n`), one can go with the default `\n`. The extra `\r` will be removed when processed. new_columns Provide an explicit list of string column names to use (for example, when scanning a headerless CSV file). If the given list is shorter than the width of the DataFrame the remaining columns will have their original name. raise_if_empty When there is no data in the source, `NoDataError` is raised. If this parameter is set to False, an empty LazyFrame (with no columns) is returned instead. truncate_ragged_lines Truncate lines that are longer than the schema. decimal_comma Parse floats using a comma as the decimal separator instead of a period. glob Expand path given via globbing rules. storage_options Options that indicate how to connect to a cloud provider. The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here: * `aws `_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \ `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. retries Number of retries if accessing a cloud instance fails. file_cache_ttl Amount of time to keep downloaded cloud files since their last access time, in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable (which defaults to 1 hour) if not given. include_file_paths Include the path of the source file(s) as a column with this name. Returns ------- LazyFrame See Also -------- read_csv : Read a CSV file into a DataFrame. Examples -------- >>> import pathlib >>> >>> ( ... pl.scan_csv("my_long_file.csv") # lazy, doesn't do a thing ... .select( ... ["a", "c"] ... ) # select only 2 columns (other columns will not be read) ... .filter( ... pl.col("a") > 10 ... ) # the filter is pushed down the scan, so less data is read into memory ... .head(100) # constrain number of returned results to 100 ... ) # doctest: +SKIP We can use `with_column_names` to modify the header before scanning: >>> df = pl.DataFrame( ... {"BrEeZaH": [1, 2, 3, 4], "LaNgUaGe": ["is", "hard", "to", "read"]} ... ) >>> path: pathlib.Path = dirpath / "mydf.csv" >>> df.write_csv(path) >>> pl.scan_csv( ... path, with_column_names=lambda cols: [col.lower() for col in cols] ... ).collect() shape: (4, 2) ┌─────────┬──────────┐ │ breezah ┆ language │ │ --- ┆ --- │ │ i64 ┆ str │ ╞═════════╪══════════╡ │ 1 ┆ is │ │ 2 ┆ hard │ │ 3 ┆ to │ │ 4 ┆ read │ └─────────┴──────────┘ You can also simply replace column names (or provide them if the file has none) by passing a list of new column names to the `new_columns` parameter: >>> df.write_csv(path) >>> pl.scan_csv( ... path, ... new_columns=["idx", "txt"], ... schema_overrides=[pl.UInt16, pl.String], ... ).collect() shape: (4, 2) ┌─────┬──────┐ │ idx ┆ txt │ │ --- ┆ --- │ │ u16 ┆ str │ ╞═════╪══════╡ │ 1 ┆ is │ │ 2 ┆ hard │ │ 3 ┆ to │ │ 4 ┆ read │ └─────┴──────┘ Nrgz(expected 'schema_overrides' dict, found zIcannot set both `with_column_names` and `new_columns`; mutually exclusivecols list[str]rac|t|tkr|tdzSS)N)r)rr7s rprz#scan_csv..with_column_names*s@4yy3{++++"T#k*:*:*<*<%===""rrr8Frcr:Trrc0g|]}t|dSrrrs rprqzscan_csv..6s3   FL v5 A A A   rrrrr5r9r;r<r+r=r>r?r@rrrDrFrHrIrLrGr.r0rArNrOrPrQrRrrKrrr)rrrar)rrrrrrrrrXrrrrr)%rSr5r8r9r:r;r<r=r+r>r?r@rrrCrDrFrGrHrIrLr.r0rArNr7rOrPrQrRrKrrrrrcredential_provider_builders% ` rprrsGN#J4*--#Bnn #:&6AA#b9LM]9^9^bbnn  #  "]CS// !  H +;X F F H#C 5E$F$FGG  # # # # # #  YUCCCC jtDDDD&3+&& #FFFF 5 9 9 9   PV     "CV_j##  " " " :" )" &~ "  : "  ) " :" *)" v"  K" &B%A" $m" e" ,+" 0/"  v!" ":#" $%" & 65'" ()" *&~+" ,*)-" .(/" 01" 2&~3" 4435" 6$m7" 8T9" :;" <(=" >87?" @&~A" B.-C" "rr) r5r8r9r:r;r<r=r+r>r?r@rrrDrFrGrHrIrLr.r0rArNrOrPrQrRrKrrrrjstr | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]] | list[bytes] CredentialProviderBuilder | Nonec d}!|ft|tsd}"t|"g}!|D])\}#}$|!|#t |$f*t | }%t|tr|}&d}ng}&|r"t|}nd}tj ||&fid|d|d| d|d|d|d| d |!d |d |d |d |%d| d|d| d|d|d|dt||d|d|d|d|d|d|d|d|d|d|d|d | }'t|'S)!Nz.expected 'schema_overrides' dict, found 'list'r8r5r@r;r<rFroverwrite_dtyperHr9r:r>r?rDwith_schema_modifyrIrLrG row_indexrArNrOrPrQrRr= cloud_optionsrrrr) rrrrrrrrr! new_from_csvrr)(rSr5r8r9r:r;r<r=r+r>r?r@rrrDrFrGrHrIrLr.r0rArNrOrPrQrRrKrrrrrrrrrsourcespylfs( rprrgsX;?J#*D11 !BCC..  $**,, 8 8DAq   q"21"5"56 7 7 7 70==&$446677  #" " " )" : "  $m "  ) " :" v" e" # " :" &~" :" *)" &B%A"  0/!" "-,#" $%" & 65'" ()" *'~7GHHH+" ,(-" ./" 0&~1" 2433" 4$m5" 6T7" 8v9" :&o;" <0/=" >?" @&~A" B.-C" DF D>>rr)FrSrTr5rUr6rVr7rWr8rXr9rYr:rYr;rZr<rZr=r[r+r\r>r]r?rUr@rUrArUrBr^rCrUrDr^rErZrFr^rGr_rHrUrIrUrJrUrKr`rLrZr.rYr0rZrMrZrNrXrOrUrPrUrQrUrRrUrar#)>rSrr5rUr6rVr8rXr9rYr:rYr;rZr<rZr=rr+rr>r]r?rUr@rUrArUrBr^rDr^rErZrFr^rGr%rHrUrIrUrLrZr.rYr0rZrMrZrNrXrOrUrPrUrQrUrRrUrar#):rSrr5rUr6rVr7rWr8rXr9rYr:rYr;rZr+r\r>r]r?rUr@rUrArUrBr^rDr^rErZrFr^rGr_rHrUrIrUrLrZr.rYr0rZrMrZrNrXrOrUrPrUrQrUrar)HrSrr5rUr8rXr9rYr:rYr;rZr<rZr=r[r+rr>r]r?rUr@rUrrUrrrCrUrDr^rFr^rGr%rHrUrIrUrLrZr.rYr0rZrArUrNrXr7rWrOrUrPrUrQrUrRrUrKr`rrrrZrr^rrYrar$)DrSrr5rUr8rXr9rYr:rYr;rZr<rZr=r[r+r[r>r]r?rUr@rUrrUrrrDr^rFr^rGr%rHrUrIrUrLrZr.rYr0rZrArUrNrXrOrUrPrUrQrUrRrUrKr`rrrrZrr^rrYrar$)G __future__r contextlibrcollections.abcriorrpathlibrtypingrr r r r polars._reexport _reexportrpolars.functions functionsrpolars._utils.deprecationr polars._utils.variousrrrrrpolars._utils.wraprrpolars.datatypesrrrpolars.io._utilsrrrr,polars.io.cloud.credential_provider._builderrpolars.io.csv._utilsrrpolars.io.csv.batched_readerrsuppress ImportError polars.polarsr r!r"rr#r$polars._typingr%r&r'polars.io.cloudr(r)rrrrrrtrrrprs"""""" $$$$$$ <<<<<<<<<<<<<<AAAAAA10000000FFFFFFFFFF FEEEEEEE999999Z%%7766666666777777777777777W''''''++++++++FFFFFFFFFF::::::VVVVVVX'99MMM-/?RRR/1CXVVV48(,!%  $ ?C).! &5"(-1"#!%"'KSSSSSWVSRNMSr48!%  $GK?C).! &5""#!%"'?OOOOOOdX'99MMM-/?RRR/1CXVVV48(,!%  ?C).! &5"("#!%"'?sssssWVSRNMsl X'99MMM-/?RRR/1CXVVV!%  $EI?C).AE&5""#!%!(,"'-1OU!%%)_OOOOOWVSRNMOz !%  $*.?C).AE&5""#!%!"&-1<@!%%)Ueeeeeeees B((B,/B,