hMhadZddlmZmZmZddlZddlZddlZ ddl m Z m Z ddl mZmZddlmZddlmZdd lmZmZmZdd lmZmZmZdd lmZdd lm Z m!Z!d e"fdZ#dee$ej%fdee$e$e&ffdZ' d3dej%de(dej%de&fdZ) d4de j*de(de(de(de(de+de j*fdZ,de-de j*de(dee$e$e&ffdZ. d5d"e j*d#ee/d$ee+e/ej%e"fd%e(de(de(de(de(d&e(de+de j*fd'Z0 d4de j*de(de(de(de(de+deej%ej%ffd(Z1 d5d"e j*d#e/d$ee+e/ej%e"fd%e(de(de(de(de(d&e(de+deej%ej%ffd)Z2 d6d*eej%e j3fd+eej%e j3fd,ee$e/fd$ee+e"e/ej%fd%e(de(de(de(de(de&fd-Z4 d7d*eej%e j3fd+eej%e j3fde(de(de(de(de&fd.Z5 d4d/e j*d0e j*de(de(de(de(de+de j*fd1Z6de-d/e j*d0e j*de(dee$e$e&ff d2Z7dS)8aiProject: PhiK - correlation analyzer library Created: 2018/09/05 Description: Functions for the Phik correlation calculation Authors: KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE. )TupleUnionOptionalN)Paralleldelayed)invpinv) definitions)phik_from_chi2),get_chi2_using_dependent_frequency_estimatesestimate_simple_ndofget_pearson_chi_square)!create_correlation_overview_tablebin_data auto_bin_data)dq_check_hist2d)array_like_to_dataframemake_shapes_equal hist_dictc|D]?\}}t|tjs |jddkrt d@t |}||}|t}| }t|}|S)a.Correlation matrix of bivariate gaussian using spark parallelization over variable-pair 2d histograms See spark notebook phik_tutorial_spark.ipynb as example. Each input histogram gets converted into correlation coefficient of bivariate gauss with correlation value rho, assuming giving binning and number of records. Correlation coefficient value is between 0 and 1. :param spark_context: spark context :param hist_dict: dict of 2d numpy grids with value-counts. keys are histogram names. :return: phik correlation matrix r z4hist_dict should be a dictionary of 2d numpy arrays.) items isinstancenpndarrayshape TypeErrorlist parallelizemap_phik_from_rowcollectr) spark_contextrkv hist_listhist_rddphik_rdd phik_list phik_overviews I/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/phik/phik.py"spark_phik_matrix_from_hist2d_dictr-#s!!TT1!RZ(( TQWQZ1__RSS SY__&&''I((33H||N++H  ""I5i@@M rowreturnc|\}}|d}t|dkr#|d|dkr|d|ddfS t|}n#t$rtj}YnwxYw|d|d|fS)zHelper function for spark parallel processing :param row: rdd row, where row[0] is key and rdd[1] :return: union of key, phik-value :rrr ?)splitlenphik_from_hist2drrnan)r/keygridc phik_values r,r"r"=sIC #A 1vv{{qtqt||tQqT3%d++ V  Q41z !!sA""A;:A;Tobservednoise_correctionexpectedcRt|tjr|j}t|tjr|j}|t |dnt ||}|rt |nd}|dkrd}t||g|j Rd|iS)a correlation coefficient of bivariate gaussian derived from chi2-value Chi2-value gets converted into correlation coefficient of bivariate gauss with correlation value rho, assuming giving binning and number of records. Correlation coefficient value is between 0 and 1. Bivariate gaussian's range is set to [-5,5] by construction. :param observed: 2d-array observed values :param bool noise_correction: apply noise correction in phik calculation :param expected: 2d-array expected values. Optional, default is None, otherwise evaluated automatically. :returns float: correlation coefficient phik Npearson)lambda_rpedestal) rpd DataFramevaluesr rrr sumr)r<r=r>chi2rBs r,r6r6Os"(BL))#?(BL))#?   5XyQQQQ #Hh 7 7 2BH#H---qH!|| $  S S S S( S SSr. data_binneddropnadrop_underflow drop_overflownjobsc|s,tjtjd|r,tjtjd|r,tjtjdj}|dkr-fdtj jj dD}nCt|fdtj jj dD}t|dkr!tjtj|| St|}|| }|| }|S) a Correlation matrix of bivariate gaussian derived from chi2-value Chi2-value gets converted into correlation coefficient of bivariate gauss with correlation value rho, assuming giving binning and number of records. Correlation coefficient value is between 0 and 1. Bivariate gaussian's range is set to [-5,5] by construction. :param pd.DataFrame data_binned: input data where interval variables have been binned :param bool noise_correction: apply noise correction in phik calculation :param bool dropna: remove NaN values with True :param bool drop_underflow: do not take into account records in underflow bin when True (relevant when binning a numeric variable) :param bool drop_overflow: do not take into account records in overflow bin when True (relevant when binning a numeric variable) :param int njobs: number of parallel jobs used for calculation of phik. default is -1. 1 uses no parallel jobs. :return: phik correlation matrix Tinplacer c Xg|]&}t|t|'S) _calc_phikr.0corIr=s r, z)phik_from_rebinned_df..sA    r;tBxx02B C C   r.rn_jobsc3|K|]6}tt|t|V7dSN)rrSrrTs r, z(phik_from_rebinned_df..sY+ +  GJ  KR$9;K L L+ + + + + + r.rindexcolumnsr_r^)replacerr7defsNaNUFOFr_ itertoolscombinations_with_replacementrErr5rCrDrreindex) rIr=rJrKrLrM column_orderr*r+s `` r,phik_from_rebinned_dfrkus8 < BFDHd;;;;DGRVT:::;DGRVT:::&L zz     =#*A   +HE***+ + + + + =#*A+ + +    9~~|BF, MMMM5i@@M")),)??M!)) )==M r.combc|\}}||kr||dfS|||gd|dt fddDr||t jfSj _tj |}|||fS)aNSplit off calculation of phik for parallel processing :param tuple comb: union of two string columns :param pd.DataFrame data_binned: input data where interval variables have been binned :param bool noise_correction: apply noise correction in phik calculation :return: tuple of variable-x, variable-y, phik-value r3F)r<rc"g|] }|jv SrRr)rUr&datahists r,rWz_calc_phik.. 0 0 0AA  0 0 0r.rr r=) groupbycountto_frameunstackfillnaanyrr7r_ droplevelr6rE)rlrIr=c0c1 phikvaluerps @r,rSrSsFB Rxx2s{""B8e"<phik_observed_vs_expected_from_rebinned_df..%sU    "JtBxx(*T"XX*>@P     r.rrXc 3K|]J}tt|t|t|VKdSr[)rrrrs r,r\z=phik_observed_vs_expected_from_rebinned_df...sn+ +  +G) * *JtBxx(*T"XX*>@P  + + + + + + r.rr]r`ra)rrrrCrDsetr_rbr7rcrdrerfrgrhrErr5rri) rrr=rJrKrLrMrjr*r+s ``` r,*phik_observed_vs_expected_from_rebinned_dfrs~8*bj)).\*-- *bj)).\*-- z! " "c**<&=&= = = = = ; 2648T:::2648T::::47BFD99947BFD999:47BFD99947BFD999%L zz       =")1    +HE***+ + + + + +  =")1 + + +    9~~|BF, MMMM5i@@M")),)??M!)) )==M r.c|\}}||kr||dfS|||g|dt fddDr||t jfS|||g|dt fddDr||t jfStt|}|||fS)aSplit off calculation of phik for parallel processing :param tuple comb: union of two string columns :param pd.DataFrame obs_binned: observed data where interval variables have been binned :param pd.DataFrame exp_binned: expected data where interval variables have been binned :param bool noise_correction: apply noise correction in phik calculation :return: r3rc"g|] }|jv SrRro)rUr&r<s r,rWz)_calc_phik_obs_vs_exp..Yrqr.rrc"g|] }|jv SrRro)rUr&r>s r,rWz)_calc_phik_obs_vs_exp.._rqr.)r<r=r>) rtrurvrwrxryrr7rr6) rlrrr=r{r|r;r>r<s @@r,rrCshFB Rxx2s{!!2r(++B/5577@@BBJJLLSSTUVVH 0 0 0 0 0 0 0112rv~!!2r(++B/5577@@BBJJLLSSTUVVH 0 0 0 0 0 0 0112rv~!844H!,rs  *)))))))))$$$$$$$$""""""""$$$$$$%%%%%% POOOOOOOOO))))))========4"c2:o."5c53I""""&QU#T#Tj#T,0#TCE:#T #T#T#T#TP" AAAA A A  A  A\AAAAH  l>B 3U?8%)/1!44 4D>4 T2:t+ ,4 4  4  4444 4\4444r" <;<;<;<; <; <;  <;  <; 2:rz !"<;<;<;<;B/1!>> >> T2:t+ ,> >  >  >>>> > 2:rz !">>>>H"&/1!22 RZ "#2 RZ "#2CI2 T4+ , 2  2  2 222 2222p" 0G0G RZ "#0G RZ "#0G0G  0G  0G  0G 0G0G0G0Gl"NN N NN  N  N  N N\NNNNb& & & & &  3U? &&&&&&r.