z-PhL UddlmZddlmZmZddlmZmZmZddl Z ddl Z ddl m ZddlmZeZeZeZeje je je je jdeje je je je jdeje je je j dej!e j"e jdej#d e j$iiZ%d e&d <d2d3dZ'd2d4dZ( d2d5dZ) d2d6dZ* d2d6dZ+ d2d7dZ,dZ-dZ. d8d9d)Z/ d8d:d/Z0 d8d;d1Z1dS)<) annotations)AnyTuple) DtypeKind ColumnBuffersColumnNullTypeN)Dtype) @)r r r )r r zdict[DtypeKind, dict[int, Any]]_PYARROW_DTYPESTdfDataFrameObjectreturnpa.Tablec6t|tjr|St|tjr tj|gSt |dst dt|||S)a. Build a ``pa.Table`` from any DataFrame supporting the interchange protocol. Parameters ---------- df : DataFrameObject Object supporting the interchange protocol, i.e. `__dataframe__` method. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Table Examples -------- >>> import pyarrow >>> from pyarrow.interchange import from_dataframe Convert a pandas dataframe to a pyarrow table: >>> import pandas as pd >>> df = pd.DataFrame({ ... "n_attendees": [100, 10, 1], ... "country": ["Italy", "Spain", "Slovenia"], ... }) >>> df n_attendees country 0 100 Italy 1 10 Spain 2 1 Slovenia >>> from_dataframe(df) pyarrow.Table n_attendees: int64 country: large_string ---- n_attendees: [[100,10,1]] country: [["Italy","Spain","Slovenia"]] __dataframe__z#`df` does not support __dataframe__) allow_copy) isinstancepaTable RecordBatch from_batcheshasattr ValueError_from_dataframer)rrs b/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/pyarrow/interchange/from_dataframe.pyfrom_dataframer ?sT"bh+ B ' '+x$$bT*** 2 ' '@>??? 2++z+BB&0 2 2 22cg}|D]'}t||}||(|s$t|}||tj|S)a Build a ``pa.Table`` from the DataFrame interchange object. Parameters ---------- df : DataFrameObject Object supporting the interchange protocol, i.e. `__dataframe__` method. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Table ) get_chunksprotocol_df_chunk_to_pyarrowappendrrr)rrbatcheschunkbatchs rrrus"G,UJ??u ,R00u 8  ) ))r!rboolpa.RecordBatchci}|D] }t|tstd|d||vrtd|d||}|jd}|t jt jt j t j t j fvrt||||<|t j krt||||<|t jkrt!||||<t#d|dt$j|S)a Convert interchange protocol chunk to ``pa.RecordBatch``. Parameters ---------- df : DataFrameObject Object supporting the interchange protocol, i.e. `__dataframe__` method. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.RecordBatch zColumn z is not a stringz is not uniquerz Data type z not handled yet) column_namesrstrrget_column_by_namedtyperINTUINTFLOATSTRINGDATETIMEcolumn_to_arrayBOOLbool_column_to_array CATEGORICAL categorical_column_to_dictionaryNotImplementedErrorrr from_pydict)rrcolumnsnamecolr/s rr$r$sJ*$&G!!LL$$$ ?=t===>> > 7??;t;;;<< <##D)) !   M N O        ,C< % %g . ..r!r> ColumnObjectpa.Arrayc|}|j}t||||j|j|}|S)a Convert a column holding one of the primitive dtypes to a PyArrow array. A primitive type is one of: int, uint, float, bool (1 bit). Parameters ---------- col : ColumnObject allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Array ) get_buffersr/buffers_to_arraysize describe_nulloffset)r>rbuffers data_typedatas rr5r5sM&ooG I GYHHJJ-J&  ( (D Kr!cH|}|ddd}|dkr|std|j}t||||j|j}|dkr&tj|tj }|S)aD Convert a column holding boolean dtype to a PyArrow array. Parameters ---------- col : ColumnObject allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Array rIrr zfBoolean column will be casted from uint8 and a copy is required which is forbidden by allow_copy=False) rB RuntimeErrorr/rCrDrErFpccastrbool_)r>rrGrDrHrIs rr7r7s$ooG 6?1 a D qyyy A    I GYHHJJ-J ( (D qyywtRXZZ(( Kr!pa.DictionaryArraycn|std|j}|dstd|d}t|}|}|d\}}t ||||j|j}tj ||} | S)aV Convert a column holding categorical data to a pa.DictionaryArray. Parameters ---------- col : ColumnObject allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.DictionaryArray zjCategorical column will be casted from uint8 and a copy is required which is forbidden by allow_copy=False is_dictionaryz-Non-dictionary categoricals not supported yet categoriesrI) rKdescribe_categoricalr:r5rBrCrDrErFrDictionaryArray from_arrays) r>r categorical cat_column dictionaryrG_rHindices dict_arrays rr9r9s$   A   *K  '=! ;== =\*J ,,JooG6?LAyw "xxzz"0"z++G #//DDJ r!ctjd|}|r9|d|d}}|dkr|dz }||fStd|)z4Parse datetime `format_str` to interpret the `data`.zts([smun]):(.*)rsz DateTime kind is not supported: )rematchgroupr:) format_strtimestamp_metaunittzs rparse_datetime_format_strrf5swX0*==N!''**N,@,@,C,Cb 3;; CKDRx MMM N NNr!c |\}}}}|tjkr(t|\}}tj||St |i|d}|r|Std|d)z+Map column date type to pyarrow date type. )reNzConversion for  is not yet supported.)rr4rfr timestamprgetr:)rHkind bit_widthf_stringrYrdrepa_dtypes r map_date_typeroGs#, D)Xq y!!!,X66b|DR(((("&&tR0044YEE  EO%C)CCCEE Er!rGrrHTuple[DtypeKind, int, str, str]lengthintrErrFc|d\}} |d\}} n#t$rd}YnwxYw |d\} } n#t$rd} YnwxYwtj|j|j|} |rt || ||||} nt | |||||} t|}| r| \}}}}tj| j| j| }|ddkrtj}n-|dkrtj}ntj }tj ||| || g| }n%tj ||| | g| }|S) a$ Build a PyArrow array from the passed buffer. Parameters ---------- buffer : ColumnBuffers Dictionary containing tuples of underlying buffers and their associated dtype. data_type : Tuple[DtypeKind, int, str, str], Dtype description of the column as a tuple ``(kind, bit-width, format string, endianness)``. length : int The number of values in the array. describe_null: ColumnNullType Null representation the column dtype uses, as a tuple ``(kind, value)`` offset : int, default: 0 Number of elements to offset from the start of the buffer. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Array Notes ----- The returned array doesn't own the memory. The caller of this function is responsible for keeping the memory owner object alive as long as the returned PyArrow array is being used. rIvalidityNoffsetsbaser]Ur rF) TypeErrorrforeign_bufferptrbufsizevalidity_buffer_from_maskvalidity_buffer_nan_sentinelro large_stringstringArray from_buffers)rGrHrqrErFr data_buffrY validity_buffvalidity_dtype offset_buff offset_dtypedata_pa_buffervalidity_pa_buff data_dtypeoffset_bit_widthoffset_pa_buffer string_typearrays rrCrCYsP6?LIq(/ (;% ~~  $+I$6! \\  &y}i6G,5777N D4]5C5B5;5;5? AA88A8E8>8>8B DDy))J $0! Q,[_-8-@2=??? Q<3  /++KK2%% o//  ikk %%   / @ &  %%   ~ . &   Ls  (( 8 AAr BufferObjectrr pa.BuffercV|\}}|\}} } } |tjksJ|tjkrdS|tjks|tjkr|dkr t j|j|j |} |tjkrn|stdt j t j |d| g|} tj| t j} n6t j t j|d| g|} |dkrtj| } | dS|tjkr'|dkr!t j|j|j |St'|d)a Build a PyArrow buffer from the passed mask buffer. Parameters ---------- validity_buff : BufferObject Tuple of underlying validity buffer and associated dtype. validity_dtype : Dtype Dtype description as a tuple ``(kind, bit-width, format string, endianness)``. describe_null : ColumnNullType Null representation the column dtype uses, as a tuple ``(kind, value)`` length : int The number of values in the array. offset : int, default: 0 Number of elements to offset from the start of the buffer. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Buffer NrrvYTo create a bitmask a copy of the data is required which is forbidden by allow_copy=Falseryr* null representation is not yet supported.)rr6r NON_NULLABLE USE_BYTEMASK USE_BITMASKrr{r|r}rKrrint8rLrMrNinvertrGr:) rrrErqrFr null_kind sentinel_val validity_kindrYbuffmask mask_bools rr~r~sB,I|+M1a IN * * * *N///t n1 1 1^///LA4E4E !2!.!6&3555 3 3 3 "F8((F*.06)88Dbhjj11II--bhjj&/3Tl5;.==I 1   ),,I  ""1%% n0 0 0\Q5F5F !2!.!6&3555 5" H H HJJ Jr!rcD|\}}}}t|} |\} } | tjkr|std|tjkr|dkrt |d| dtj | |d|g|} tj | } tj | } | dS| tjkr|std|tjkrtj}n| }tj ||d|g|} tj| | }tj |}| dS| tjkrdSt |d) a Build a PyArrow buffer from NaN or sentinel values. Parameters ---------- data_pa_buffer : pa.Buffer PyArrow buffer for the column data. data_type : Dtype Dtype description as a tuple ``(kind, bit-width, format string, endianness)``. describe_null : ColumnNullType Null representation the column dtype uses, as a tuple ``(kind, value)`` length : int The number of values in the array. offset : int, default: 0 Number of elements to offset from the start of the buffer. allow_copy : bool, default: True Whether to allow copying the memory to perform the conversion (if false then zero-copy approach is requested). Returns ------- pa.Buffer rr z with rhNryrr)rorUSE_NANrKrr2r:rrrrLis_nanrrG USE_SENTINELr4int64equalr)rrHrErqrFrrkrlrYrrr pyarrow_datarsentinel_dtype sentinel_arrrs rrrsB&D)Qy))J+I|N*** B  9? " "yB&EEIEEEGG G800~& 1L 9\**D9T??D<<>>!$ $ n1 1 1 B  9% % %XZZNN'Nx,,^-3.2N-C4:-<< x l;; Il++   ""1%% n1 1 1 ! H H HJJ Jr!)T)rrrr)rr)rrrr)rr*)r>r?rr)rr@)r>r?rr)rrO)rT)rGrrHrprqrrrErrFrrrr)rr@)rrrr rErrqrrrFrrrr)rr)rrrHr rErrqrrrFrrrr)rr)2 __future__rtypingrrpyarrow.interchange.columnrrrpyarrowrr_pyarrow.computecomputerLr rr?rr0rint16int32rr1uint8uint16uint32uint64r2float16float32float64r6rNr3rr__annotations__r rr$r5r7r9rfrorCr~rr!rrs$#""""""   ,,,,,,   Mwrwyy      $$N """&&O*"*,,$"*,,$"*,,((N   $$ q)")++&4$3232323232l*****>,/,/,/,/,/b>$$$$$R-----`OOO$EEE. fffff\ LJLJLJLJLJh UJUJUJUJUJUJUJr!