z-Ph} .ddlZddlZddlZddlZddlmZddlm Z ddlm Z m Z  ddl m Z n #e$rdZ YnwxYwejj Zd*dZdZejddd gd Zejd ejd d dfdZdZejddd gdZdZdZejddd gdZdZdZejddd gdZdZejddd gdZ ejj!dZ"ejj!dZ#ejdej$ej%ddej$ej%dej%dej%dd kgd!Z&d"Z'd#Z(d$Z)d%Z*d&Z+d'Z,d(Z-d)Z.dS)+N)tobytes) ArrowInvalidArrowNotImplementedError cHddlm}|tj|S)Nr)_get_udf_context)pyarrow._computerpadefault_memory_pool) batch_lengthrs \/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/pyarrow/tests/test_substrait.pymock_udf_contextr&s0111111  B244l C CCctjt||}tj||j5}||dddn #1swxYwY|S)N)schema) ospathjoinstrr ipcRecordBatchFileWriterr write_table)tmpdir file_nametablerwriters r _write_dummy_data_to_diskr+s 7<<F Y / /D  % %d5< % @ @"F5!!!""""""""""""""" KsA66A:=A: use_threadsTFcd}d}tjgdgdg}t|||}t|dt j|}tj |}tj ||}| } | dg| dgksJdS)Nax { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "foo" ] }, "local_files": { "items": [ { "uri_file": "FILENAME_PLACEHOLDER", "arrow": {} } ] } } }} ] } zread_data.arrowfoonamesFILENAME_PLACEHOLDERrr rrrreplacepathlibPathas_uri _substrait_parse_json_plan substrait run_queryread_allselect rrsubstrait_queryrrrquerybufreaderres_tbs r test_run_serialized_queryr<2sO<"I Hooo&ug 6 6 6E $VY > >D O++ T 2 2 9 9 ; ;== > >E - ( ( / /C  + > > >F __  F << FMM5'$:$: : : : : : :rr8sbuffersbytesr!ct|tjtfs\dt |d}t jt|5tj |dddn #1swxYwYdSd}t jt|5tj |ddddS#1swxYwYdS)Nz)Expected 'pyarrow.Buffer' or bytes, got ''matchz1ParseFromZeroCopyStream failed for substrait.Plan) isinstancer Bufferbytestypepytestraises TypeErrorr2r3r)rr8msgs r test_run_query_input_typesrI`sD ebi/ 0 0H$u++HHH ]9C 0 0 0 ' '   & & & ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' >C |3 / / /##E"""##################s$A11A58A5B==CCcd}tjt|}d}t jt |5tj|ddddS#1swxYwYdS)Nz2 { "relations": [ ] } zPlan has no relationsr?) r r0r1rrErFrr2r3)r8r9 exec_messages r test_invalid_planrLps E - ( ( 8 8C*L |< 8 8 8!!C   !!!!!!!!!!!!!!!!!!s A..A25A2cd}d}tjgdgdg}t|||}t|dt j|}tj t|}tj ||}| } | dg| dgksJdS)Na { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "bar" ] }, "local_files": { "items": [ { "uri_file": "FILENAME_PLACEHOLDER", "arrow": {}, "metadata" : { "created_by" : {}, } } ] } } }} ] } zbinary_json_data.arrowr barr'r)r*r+r6s r (test_binary_conversion_with_json_optionsrO}sOB)I Hooo&ug 6 6 6E $VY > >D O++ T 2 2 9 9 ; ;== > >E - ( ( 8 8C  + > > >F __  F << FMM5'$:$: : : : : : :rcN|d|}|D]}||rdSdS)N#TF)endswith)fnsext_filefn_namesuffixfns r has_functionrXsI $ $7 $ $F ;;v   44  5rctj}t|ddsJt|ddsJdS)Nzfunctions_arithmetic.yamladdsum)r r0get_supported_functionsrX)supported_functionss r test_get_supported_functionsr^sj-??AA +3U < <<<< +3U < <<<<<.table_providersd 7/00 0 1X  X%%%%  1X   566 6ra { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "x" ] }, "namedTable": { "names": ["t1"] } } }} ] } rmr) r Table from_pydictrfieldint64r0r1rr2r3r4) rrmr7r9r:r;rjrkrls @@@r test_named_tablerss8''iii(899L8''iii(899Ly"(3 33455H 7 7 7 7 7 7 7O2 - ( ()A)A B BC \ # # N $EEF __  F \ ! ! ! ! ! !rcLtjdgdifd}d}tjt |}d}t jt|5tj ||ddddS#1swxYwYdS)Nr`rac`|std|ddkrStdNrdrrergrhr(_rks r rmz;test_named_table_invalid_table_name..table_provider> 7/00 0 1X   566 6ra { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "x" ] }, "namedTable": { "names": ["t3"] } } }} ] } zInvalid NamedTable Sourcer?rm) r rorpr0r1rrErFrr2r3)rmr7r9rKrks @r #test_named_table_invalid_table_namer{s8''iii(899L77777O2 - ( ()A)A B BC.L |< 8 8 8@@C????@@@@@@@@@@@@@@@@@@s5BB Bcjtjdgdifd}d}t|}tjt|}d}t jt|5tj ||ddddS#1swxYwYdS)Nr`rac`|std|ddkrStdrvrhrws r rmz4test_named_table_empty_names..table_providerryra { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "x" ] }, "namedTable": { "names": [] } } }} ] } z!names for NamedTable not providedr?rz) r rorprr0r1rErFrr2r3)rmr7r8r9rKrks @r test_named_table_empty_namesr~s8''iii(899L77777O0 O $ $E - ( ( 8 8C6L |< 8 8 8@@C????@@@@@@@@@@@@@@@@@@sB((B,/B,c  tjdgdi fd}d}tj|}tj|||}|}|\}} dd|td d} || ksJdS) Nr`rac`|std|ddkrStdrvrhr(rx test_tables r rmz.test_udf_via_substrait..table_providerJ> 7/00 0 1X   566 6rs { "extensionUris": [ { "extensionUriAnchor": 1 }, { "extensionUriAnchor": 2, "uri": "urn:arrow:substrait_simple_extension_function" } ], "extensions": [ { "extensionFunction": { "extensionUriReference": 2, "functionAnchor": 1, "name": "y=x+1" } } ], "relations": [ { "root": { "input": { "project": { "common": { "emit": { "outputMapping": [ 1, 2, ] } }, "input": { "read": { "baseSchema": { "names": [ "t", ], "struct": { "types": [ { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, ], "nullability": "NULLABILITY_REQUIRED" } }, "namedTable": { "names": [ "t1" ] } } }, "expressions": [ { "selection": { "directReference": { "structField": {} }, "rootReference": {} } }, { "scalarFunction": { "functionReference": 1, "outputType": { "i64": { "nullability": "NULLABILITY_NULLABLE" } }, "arguments": [ { "value": { "selection": { "directReference": { "structField": {} }, "rootReference": {} } } } ] } } ] } }, "names": [ "x", "y", ] } } ] } rnr!yr) r rorpr0r1r2r3r4 add_columnr) unary_func_fixturerrmr7r9r:r;functionname expected_tbrs @r test_udf_via_substraitrFs%%sIII&677J77777cOJ - ( ( 9 9C \ # # N $EEF __  F'NHd''3jo1/1/00K [ rcxtjdgdifd}d}tj|}t jtj5}tj ||dddn #1swxYwYdt|j vsJdS)Nr`rac`|std|ddkrStdrvrhrs r rmz=test_udf_via_substrait_wrong_udf_name..table_providerrrs { "extensionUris": [ { "extensionUriAnchor": 1 }, { "extensionUriAnchor": 2, "uri": "urn:arrow:substrait_simple_extension_function" } ], "extensions": [ { "extensionFunction": { "extensionUriReference": 2, "functionAnchor": 1, "name": "wrong_udf_name" } } ], "relations": [ { "root": { "input": { "project": { "common": { "emit": { "outputMapping": [ 1, 2, ] } }, "input": { "read": { "baseSchema": { "names": [ "t", ], "struct": { "types": [ { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, ], "nullability": "NULLABILITY_REQUIRED" } }, "namedTable": { "names": [ "t1" ] } } }, "expressions": [ { "selection": { "directReference": { "structField": {} }, "rootReference": {} } }, { "scalarFunction": { "functionReference": 1, "outputType": { "i64": { "nullability": "NULLABILITY_NULLABLE" } }, "arguments": [ { "value": { "selection": { "directReference": { "structField": {} }, "rootReference": {} } } } ] } } ] } }, "names": [ "x", "y", ] } } ] } rzzNo function registered) r rorpr0r1rErF ArrowKeyErrorr2r3rvalue)rmr7r9excinfors @r %test_udf_via_substrait_wrong_udf_namers %%sIII&677J77777cOJ - ( ( 9 9C r' ( (CG s>BBBCCCCCCCCCCCCCCC #s7='9'9 9 9 9 9 9 9s)"BBBcvtjdgdifd}d}tjt |}tj|||}|}tjdgdi}||ksJdS)Nr`racSN)r(rin_tables r rmz/test_output_field_names..table_provider<sra { "version": { "major": 9999 }, "relations": [ { "root": { "input": { "read": { "base_schema": { "struct": { "types": [{"i64": {}}] }, "names": ["x"] }, "namedTable": { "names": ["t1"] } } }, "names": ["out"] } } ] } rnout) r rorpr0r1rr2r3r4)rrmr7r9r:r;expectedrs @r test_output_field_namesr8sx##S)))$455HO4 - ( ()A)A B BC \ # # N $EEF __  Fx##UIII$677H X      rcntjgdgdgddfd}d}tj|}tj||d}|}tjd d gd d gd }||ksJdS)Nr!r!r"r")r!r"r#r$)?rrr)kv1v2cSrrrs r rmz7test_scalar_aggregate_udf_basic..table_providerk rs { "extensionUris": [ { "extensionUriAnchor": 1, "uri": "urn:arrow:substrait_simple_extension_function" }, ], "extensions": [ { "extensionFunction": { "extensionUriReference": 1, "functionAnchor": 1, "name": "sum_mean" } } ], "relations": [ { "root": { "input": { "extensionSingle": { "common": { "emit": { "outputMapping": [ 0, 1 ] } }, "input": { "read": { "baseSchema": { "names": [ "k", "v1", "v2", ], "struct": { "types": [ { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, { "i64": { "nullability": "NULLABILITY_NULLABLE" } }, { "fp64": { "nullability": "NULLABILITY_NULLABLE" } } ], "nullability": "NULLABILITY_REQUIRED" } }, "namedTable": { "names": ["t1"] } } }, "detail": { "@type": "/arrow.substrait_ext.SegmentedAggregateRel", "segmentKeys": [ { "directReference": { "structField": {} }, "rootReference": {} } ], "measures": [ { "measure": { "functionReference": 1, "phase": "AGGREGATION_PHASE_INITIAL_TO_RESULT", "outputType": { "fp64": { "nullability": "NULLABILITY_NULLABLE" } }, "arguments": [ { "value": { "selection": { "directReference": { "structField": { "field": 1 } }, "rootReference": {} } } }, { "value": { "selection": { "directReference": { "structField": { "field": 2 } }, "rootReference": {} } } } ] } } ] } } }, "names": [ "k", "v_avg" ] } } ], } Frnr!r"g@g@)rv_avgr rorpr0r1r2r3r4varargs_agg_func_fixturermr7r9r:r;rrs @r test_scalar_aggregate_udf_basicrcs%%ll,,,### % %J |Oz - ( ( 9 9C \ # # N$??F __  F(&&Vs((K [ rcztjgdgdgdgddfd}d}tj|}tj||d }|}tjgd gd gd d }||ksJdS)N)r!r!r!r!r"r"r"r")r!rrr!rr!rr!)r!r"r#r$r%rb)rrrrg@g@g@g@)trrrcSrrrs r rmz5test_hash_aggregate_udf_basic..table_providerrrs  { "extensionUris": [ { "extensionUriAnchor": 1, "uri": "urn:arrow:substrait_simple_extension_function" }, ], "extensions": [ { "extensionFunction": { "extensionUriReference": 1, "functionAnchor": 1, "name": "sum_mean" } } ], "relations": [ { "root": { "input": { "extensionSingle": { "common": { "emit": { "outputMapping": [ 0, 1, 2 ] } }, "input": { "read": { "baseSchema": { "names": [ "t", "k", "v1", "v2", ], "struct": { "types": [ { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, { "i64": { "nullability": "NULLABILITY_NULLABLE" } }, { "fp64": { "nullability": "NULLABILITY_NULLABLE" } } ], "nullability": "NULLABILITY_REQUIRED" } }, "namedTable": { "names": ["t1"] } } }, "detail": { "@type": "/arrow.substrait_ext.SegmentedAggregateRel", "groupingKeys": [ { "directReference": { "structField": { "field": 1 } }, "rootReference": {} } ], "segmentKeys": [ { "directReference": { "structField": {} }, "rootReference": {} } ], "measures": [ { "measure": { "functionReference": 1, "phase": "AGGREGATION_PHASE_INITIAL_TO_RESULT", "outputType": { "fp64": { "nullability": "NULLABILITY_NULLABLE" } }, "arguments": [ { "value": { "selection": { "directReference": { "structField": { "field": 2 } }, "rootReference": {} } } }, { "value": { "selection": { "directReference": { "structField": { "field": 3 } }, "rootReference": {} } } } ] } } ] } } }, "names": [ "t", "k", "v_avg" ] } } ], } Frnr)r!rrr!) @rg"@g&@)rrrrrs @r test_hash_aggregate_udf_basicrs%%&&&&&&'''777 9 9JNO^ - ( ( 9 9C \ # # N$??F __  F(&& \\ \\&&&((K [ rexprr`rr2ctjtjdtjtjdtjg}tj|gdg|}tj|}||jksJt|jdksJd|jvsJdS)Nr`r test_exprr!) r rrqint32r2serialize_expressionsdeserialize_expressionslen expressions)rrr9returneds r test_serializing_expressionsrs Y bhjj!! bhjj!!F , , ,dVk]F K KC|33C88H X_ $ $ $ $ x# $ $ ) ) ) ) (. . . . . . .rctjddftjddftjddftjdftjdftjdfd}tjd|Dfd }|D]=\}\}}|tj |tj || k>dS) Nsrmsns test_strings test_string) time_seconds time_millis time_nanos date_millis large_string large_binarycDg|]\}\}}tj||Sr)r rq).0rtyprxs r z-test_arrow_specific_types..s,QQQhsAs++QQQrctj|gdg}tj|}|jksJdSNr)r r2rrr)rr9rrs r check_round_tripz3test_arrow_specific_types..check_round_tripsNl00$+OO<77<<((((((r)rD) r time32time64date64rrritemspcrqscalar)fieldsrrrvalrs @r test_arrow_specific_typesrs3+ $+y* Q'**M:**N; FYQQ&,,..QQQ R RF))))) #LLNNEEjsC$29Ss+C+C+CCDDDDEErc ftjtjdtjtjdtjtjdtjtjtjtjdtjtjtjgtjtjdtj tjdtjtjdtjtjdtjgfd}D] }||dS)N binary_view string_view dictionaryreectjtj|j}tj|gdg}tj|}|jksJdSr) ris_nullrqrr r2rrr)rqrr9r alt_schemars r check_one_wayz/test_arrow_one_way_types..check_one_waysiz"(5:..//l00$+OO<77<<X_,,,,,,r) r rrqrrrrstringrun_end_encodedbinary)rrqrrs @@r test_arrow_one_way_typesrsO Y H]BN$4$4 5 5 H]BN$4$4 5 5 H\2=RY[[#I#I J J HUB.rxzz29;;GG H H  F H]BIKK 0 0 H]BIKK 0 0 H\29;; / / HUBIKK ( (  J------  erctjtjdtjtjdtjg}t jt jdd}t jt jdd}t jt5}tj |gg|dddn #1swxYwYdt|j vsJt jt5}tj |gddg|dddn #1swxYwYdt|j vsJt jt5}tj |gdg|dddn #1swxYwYd t|j vsJdS) Nr`rrzzneed to have the same lengthr&rNrzNo match for FieldRef) r rrqrrequalrErF ValueErrorr2rrr)rrbad_exprrs r test_invalid_expression_ser_desrsm Y bhjj!! bhjj!!F 8BHSMM1 % %Dx q))H z " "?g **D62v>>>??????????????? )S-?-? ? ? ? ? z " "Kg **D6E5>6JJJKKKKKKKKKKKKKKK )S-?-? ? ? ? ? z " "Ig **H:xHHHIIIIIIIIIIIIIII "c'-&8&8 8 8 8 8 8 8s6#C55C9<C91%E""E&)E&$GGGctjtjdtjtjdtjg}t jt jddt jt jdt jdg}tj|ddg|}tj|}||jksJt|j dksJt jt jddt jt jdt jdg}t|j dt|dksJt|j dt|dksJdS) Nr`rrfirstsecondr"rr!) r rrqrrrr2rrrrr)rexprsr9r norm_exprss r %test_serializing_multiple_expressionsrsz Y bhjj!! bhjj!!FXbhsmmQ ' '"(3--#)O)O PE , , ,UWh4G P PC|33C88H X_ $ $ $ $ x# $ $ ) ) ) )(28A;;**BHRXa[["(1++,N,NOJ x#G, - -Z]1C1C C C C C x#H- . .#jm2D2D D D D D D DrcPtjtjdtjtjdtjg}t jt jdd}t jt jdd}||}tj|}||jksJt|j dksJt|j dt|ksJtj ||gddg|}tjt5}tj|dddn #1swxYwYd t|jvsJtj |gd g|}tj|}t|t|ksJdS) Nr`rrrr! expressionrrzcontained multiple expressions weirdname)r rrqrrr to_substraitr2rrrrrrErFr Expressionfrom_substraitr)rr expr_normr9rrexpr2s r test_serializing_with_computer s Y bhjj!! bhjj!!F 8BHSMM1 % %D!a((I   F # #C|33C88H X_ $ $ $ $ x# $ $ ) ) ) ) x#L1 2 2c)nn D D D D , , ,dD\GX;NPV W WC z " "*g $$S)))*************** +s7=/A/A A A A A , , ,dVk]F K KC M ( ( - -E u::Y ' ' ' ' ' 's3 FF#&F#ctjtjdtjg}t jd}t jd}t j||g}tjt5tj |dg|dddn #1swxYwYtj |dg|d}tj |}||jksJt|jdksJt|jdt|dksJdS) Nr`rr$rT)allow_arrow_extensionsr!r)r rrquint32rr shift_leftrErFrr2rrrrr)rabrr9rs r test_serializing_udfsr$sY bikk""F " A ! A ]1a !E / 0 0DD **56(FCCCDDDDDDDDDDDDDDD , , , x - > >C|33C88H X_ $ $ $ $ x# $ $ ) ) ) ) x#F+ , ,E!H = = = = = =s#CCCctd}tjtjdtjtjdtjg}tj|}||ksJtj|}|j|ksJtj|}||ksJtj|j}||ksJtj|j }|j|ksJdS)Ns x y * br`r) r rrqrrr2deserialize_schemaserialize_schemarr)substrait_schemaexpected_schemararrow_substrait_schemas r test_serializing_schemar:sVi bhjj!! bikk""!O|../?@@H h & & & &\::8DD ! (,< < < < <|../EFFH h & & & &|../E/LMMH h & & & &|334J4UVVH ?o - - - - - -rcGdd}d}tjj||}t |jdksJd|jvsJd|jvsJdS)NceZdZdZdZdS)7test_bound_expression_from_Message..FakeMessagec||_dSrr)selfrs r __init__z@test_bound_expression_from_Message..FakeMessage.__init__Rs DIIIrc|jSrr)rs r SerializeToStringzItest_bound_expression_from_Message..FakeMessage.SerializeToStringUs 9 rN)__name__ __module__ __qualname__r r rrr FakeMessagerQs2        rrsj  project_release  project_version"0 project_version project_release : br"project_releaseproject_version)r r2BoundExpressionsrrr)rmessagers r "test_bound_expression_from_MessagerPsAG L ) 8 8W9M9M N NE u ! !Q & & & &  1 1 1 1 1  1 1 1 1 1 1 1r)r)/rr-rEpyarrowr pyarrow.computecomputer pyarrow.librrrpyarrow.substraitr2 ImportErrormark pytestmarkrr parametrizer< py_bufferrIrLrOrXr^rsr{r~rrrnumpyrrrrqrrrrrrrrrrrr r s$  >>>>>>>>)))))))III [ " DDDD u 66*;*;76*;Z<2< #:#:Ha"HII # #JI # ! ! !u 66,;,;76,;b<<<u 66-"-"76-"`'@'@'@T'@'@'@Tu 66x!x!76x!vs:s:s:lu 66''76'TQ!Q!Q!hg!g!g!T BHXRXc]]A BHXRXc]]HBHSMM** BHSMMB" / /  /EEE(8999( E E E (((6>>>,...,22222s -77