hMhzRdZddlmZmZddlZddlZddlZddl Z ddl Z ddl m Z ddl m Z mZddlmZddlmZmZdd lmZdd lmZmZdd lmZdd lmZmZdd lmZm Z  ddej"d"e#d#e)d$e)d)e)d%e#dee$e$ffd*Z, d?d,ej-d#e)d$e)d"e#d)e)d-e.d.e.d/e.d%e#dej-fd0Z/ d@d2ej-d3e!d#e)d$e)d"e#d)e)d4ee#e!ej"e0fd-e.d.e.d/e.d5e.d%e#dej-fd6Z1 dAdeej"ej2fd8eej"ej2fd4ee#e!ej"e0fd9e.d#e)d"e#d)e)d$e)d-e.d.e.d/e.d%e#dee$e$ffd:Z3 dBdeej"ej2fd8eej"ej2fd#e)d)e)d"e#d$e)d-e.d.e.d/e.d%e#dee$e$ffd;Z4dS)CaProject: PhiK - correlation analyzer library Created: 2018/09/05 Description: Functions for doing the significance evaluation of an hypothesis test of variable independence using a contingency table. Authors: KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE. )TupleUnionN)stats)specialoptimize) definitions)bin_data!create_correlation_overview_table),get_chi2_using_dependent_frequency_estimates) estimate_ndoftheoretical_ndof)sim_chi2_distribution)dq_check_nunique_valuesdq_check_hist2d)array_like_to_dataframeguess_interval_cols2chi2snbinsreturnc d  fd fd}tj| tj  t| tj||\}}|d|dz |ddtj|d z z}d }t j||d ||f }|jd fS) a5 Fit the hybrid chi2-distribution to the data to find f. Perform a binned likelihood fit to the data to find the optimal value for the fraction f in h(x|f) = N * (f * chi2(x, ndof) + (1-f) * gauss(x, ndof, sqrt(ndof)) The parameter ndof is fixed in the fit using ndof = mean(x). The total number of datapoints N is also fixed. :param list chi2s: input data - a list of chi2 values :param int nbins: in order to fit the data a histogram is created with nbins number of bins :returns: f, ndof, sigma (width of gauss), bw (bin width) c||tj||zd|z tj|||zzzSNr rchi2pdfnormxNfksigmas Q/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/phik/significance.pymyfuncz/fit_test_statistic_distribution..myfunc3sBA q!,,,A1e9T9T/TTUUc(|z|S)N)r r"bwkmeanlsigmar&nsims r%z1fit_test_statistic_distribution..6s4"9a??r'c|g|R}|tj||z tj|dzz}tj|Sr)rxlogygammalnnpsqrt)pr yr"llffuncs r%gtestz.fit_test_statistic_distribution..gtest8sM E!LaLLL q!$$ $wq1u'='= =wr{{r'binsr rN)?))g)r )boundsargs) r2meanr3len histogramdiffr least_squaresr )rrr8yhistxboundsxhist initGuessresr*r7r+r,r&r-s @@@@@@r%fit_test_statistic_distributionrJ$sVVV @ ? ? ? ? ? ? ?E GENNE WU^^F u::D\%e444NE7 gaj B CRCL277++a/ /EI   yuen   C 58UFB &&r'r r!r"r#r$c||tj||zd|z tj|||zzzS)aX Definition of the combined probability density function h(x) h(x|f) = N * (f * chi2(x, k) + (1-f) * gauss(x, k, sigma) :param float x: x :param float N: normalisation :param float f: fraction [0,1] :param float k: ndof of chi2 function and mean of gauss :param float sigma: width of gauss :return: h(x|f) r rrs r%hfuncrLMsD EJNN1a(((AEUZ^^Aq%5P5P+PP QQr'rndofcxtj||}tj| }|dkrq||z }t jdt jz |t j|zz ||dz zz}t j|t j|z }||fS)a Convert a chi2 into significance using knowledge about the number of degrees of freedom Conversion is done using asymptotic approximation. :param float chi2: chi2 value :param float ndof: number of degrees of freedom :returns: p_value, significance rr<r ) rrsfrppfmathlogpir3)rrMp_valuez_valuezus r%significance_from_chi2_ndofrX]sjmmD$''Gz~~g&&&G !|| 4K Xa$'k " " "TDHQKK%7 7$!a%. H)A O,, G r'valuescNt|}t||\}}||fS)a Convert a chi2 into significance using knowledge about the number of degrees of freedom Convention is done using asymptotic approximation. :param float chi2: chi2 value :param float ndof: number of degrees of freedom :returns: p_value, significance )rrX)rYrrMrTrUs r%!significance_from_chi2_asymptoticr[us0 F # #D24>>GW G r'log-likelihood multinominalr;r-lambda_simulation_methodnjobsc|t|||||}dtj||dz z }tj| }||fS)a Convert a chi2 into significance using knowledge about the shape of the chi2 distribution of simulated data Calculate significance based on simulation (MC method), using a simple percentile. :param float chi2: chi2 value :param list chi2s: provide your own chi2s values (optional) :param int njobs: number of parallel jobs used for simulation. default is -1. 1 uses no parallel jobs. :returns: pvalue, significance Nr-r_r`rar=gY@)rrpercentileofscorerrP) rrYr-r_r`rraempirical_p_valueempirical_z_values r%significance_from_chi2_MCrgso* }% /    e5eTBBUJJ(9::: / //r'c.|t|||||}||jdz |jdz}|dkr`t|\}} } } |tj|| zd|z tj|| | zz} n/t|} tj|| } tj |  } | dkr|| z }tj dtj z | tj |zz | |dz zz}|dkr|dtj |zz }tj |tj |z } | | fS)a Convert a chi2 into significance using a hybrid method This method combines the asymptotic method with the MC method, but applies several corrections: * use effective number of degrees of freedom instead of number of degrees of freedom. The effective number of degrees of freedom is measured as mean(chi2s), with chi2s a list of simulated chi2 values. * for low statistics data sets, with on average less than 4 data points per bin, the distribution of chi2-values is better described by h(x|f) then by the usual chi2-distribution. Use h(x|f) to convert the chi2 value to the pvalue and significance. h(x|f) = N * (f * chi2(x, ndof) + (1-f) * gauss(x, ndof, sqrt(ndof)) :param float chi2: chi2 value :param list chi2s: provide your own chi2s values (optional) :param float avg_per_bin: average number of data points per bin :param int njobs: number of parallel jobs used for simulation. default is -1. 1 uses no parallel jobs. :returns: p_value, significance Nrcrr r<)rsumshaperJrrrOrr rPrQrRrSr3)rrYr-r_r`rra avg_per_binr"endofr,r*pvalue_hzvalue_hrVrWs r%significance_from_chi2_hybridrqs< }% /    **,,a06<?BKa>uEE5&"uz}}T5111QUejmm %? ? 5  e$$:==u-- x(((H1}} 5L Xa$'k " " "UTXa[[%8 85AE? J !   dhqkk! !A9Q!_-- X r'hybridsignificance_methodc t||}|dkrt||\}}n`|dkrt||||||\}}nA|dkrt||||||\}}n"t d|||fS)a Calculate the significance of correlation of two variables based on the contingency table :param values: contingency table :param int nsim: number of simulations :param str lambda_: test statistic. Available options are [pearson, log-likelihood] :param str simulation_method: simulation method. Options: [multinominal, row_product_multinominal, col_product_multinominal, hypergeometric]. :param str significance_method: significance_method. Options: [asymptotic, MC, hybrid] :param int njobs: number of parallel jobs used for simulation. default is -1. 1 uses no parallel jobs. :return: pvalue, significance )r_ asymptoticMCrcrrz"simulation_method {0:s} is unknown)r r[rgrqNotImplementedErrorformat) rYr-r_r`rsrarpvaluezvalues r%significance_from_hist2dr{s, 8 P P PDl**:64HH  $ $2  /      ( (7  /    " 0 7 78I J J    6>r'T data_binneddropnadrop_underflow drop_overflowc |s,|tjtjd|r,|tjtjd|r,|tjtjd|j} g} ttj |jj dD]9\} \} } | | | g|  d}d|jvs d|jvrc| | | tjft'jd| |jd| |jd|j|_|j }t/||||||\}}| | | |f;t1| dkr!t3jtj| | St7| }|| }|| }|S) a Calculate significance of correlation of all variable combinations in the DataFrame :param data_binned: input binned DataFrame :param int nsim: number of simulations :param str lambda_: test statistic. Available options are [pearson, log-likelihood] :param str simulation_method: simulation method. Options: [mutlinominal, row_product_multinominal, col_product_multinominal, hypergeometric]. :param str significance_method: significance_method. Options: [asymptotic, MC, hybrid] :param bool dropna: remove NaN values with True :param bool drop_underflow: do not take into account records in underflow bin when True (relevant when binning a numeric variable) :param bool drop_overflow: do not take into account records in overflow bin when True (relevant when binning a numeric variable) :param int njobs: number of parallel jobs used for simulation. default is -1. :return: significance matrix T)inplacer<rr z[Too few unique values for variable {0:s} ({1:d}) or {2:s} ({3:d}) to calculate significance)r-r_r`rsra)indexcolumns)r)r)replacer2nandefsNaNUFOFr enumerate itertoolscombinations_with_replacementrYgroupbycountto_frameunstackfillnarlappendwarningswarnrx droplevelr{rApd DataFramer reindex)r|r_r`r-rsr}r~rra column_ordersignifsic0c1datahistryrzsignificance_overviews r%significance_from_rebinned_dfr&sb: < BFDHd;;;;DGRVT:::;DGRVT:::&LG / 0C0JANN)) 8B   R ) )" - 3 3 5 5 > > @ @ H H J J Q QRS T T    !x~"5"5 NNBBF+ , , , Mmttq)2x~a/@    #+5577?1 / 3     B'(((( 7||q|BF, MMMM=gFF299,9OO199 9MM  r' df interval_colsr:verbosec |t|| }t|||\} } t| | |}t|||||||| |  S)a Calculate significance of correlation of all variable combinations in the dataframe :param pd.DataFrame df: input data :param list interval_cols: column names of columns with interval variables. :param int nsim: number of simulations :param str lambda_: test statistic. Available options are [pearson, log-likelihood] :param str simulation_method: simulation method. Options: [mutlinominal, row_product_multinominal, col_product_multinominal, hypergeometric]. :param int nsim: number of simulated datasets :param str significance_method: significance_method. Options: [asymptotic, MC, hybrid] :param bool dropna: remove NaN values with True :param bins: number of bins, or a list of bin edges (same for all columns), or a dictionary where per column the bins are specified. (default=10) E.g.: bins = {'mileage':5, 'driver_age':[18,25,35,45,55,65,125]} :param bool dropna: remove NaN values with True :param bool drop_underflow: do not take into account records in underflow bin when True (relevant when binning a numeric variable) :param bool drop_overflow: do not take into account records in overflow bin when True (relevant when binning a numeric variable) :param bool verbose: if False, do not print all interval columns that are guessed :param int njobs: number of parallel jobs used for simulation. default is -1. 1 uses no parallel jobs. :return: significance matrix N)r}r9)r_r`r-rsr}r~rra)rrr r)rrr_r`r-rsr:r}r~rrradf_cleaninterval_cols_cleanr|s r%significance_matrixrvsL+B88 $; M&%%%!H!8%8tDDDK (+ /%#    r'Fr5quantilec |g}nt|tr|g}t|dkr0t||} t | |||jj\}}t||||||| | | |  S)a Calculate the significance of correlation Calculate the significance of correlation for two variables which can be of interval, oridnal or categorical type. Interval variables will be binned. :param x: array-like input :param y: array-like input :param num_vars: list of numeric variables which need to be binned, e.g. ['x'] or ['x','y'] :param bins: number of bins, or a list of bin edges (same for all columns), or a dictionary where per column the bins are specified. (default=10) E.g.: bins = {'mileage':5, 'driver_age':[18,25,35,45,55,65,125]} :param quantile: when bins is an integer, uniform bins (False) or bins based on quantiles (True) :param str lambda_: test statistic. Available options are [pearson, log-likelihood] :param int nsim: number of simulated datasets :param str simulation_method: simulation method. Options: [mutlinominal, row_product_multinominal, col_product_multinominal, hypergeometric]. :param str significance_method: significance_method. Options: [asymptotic, MC, hybrid] :param bool dropna: remove NaN values with True :param bool drop_underflow: do not take into account records in underflow bin when True (relevant when binning a numeric variable) :param bool drop_overflow: do not take into account records in overflow bin when True (relevant when binning a numeric variable) :param int njobs: number of parallel jobs used for simulation. default is -1. 1 uses no parallel jobs. :return: p-value, significance Nr)r:r)r_rsr-r`r}r~rra) isinstancestrrArr TrYsignificance_from_binned_array)r r5num_varsr:rr_r-rsr`r}r~rrars r%significance_from_arrayrsP Hc " ": 8}}q $Q * *H4(CCCEL1 )  / +%#    r'c 8|stj|tjt j}tj|tjt j}|s|r"tj|t j}tj|t j}|r^tj |tj |tj k<tj |tj |tj k<|r^tj |tj |tj k<tj |tj |tj k<tj ||j} t| stj tj fSt| ||||| S)ac Calculate the significance of correlation Calculate the significance of correlation for two variables which can be of interval, oridnal or categorical type. Interval variables need to be binned. :param x: array-like input :param y: array-like input :param str lambda_: test statistic. Available options are [pearson, log-likelihood] :param str simulation_method: simulation method. Options: [multinominal, row_product_multinominal, col_product_multinominal, hypergeometric]. :param int nsim: number of simulated datasets :param str significance_method: significance_method. Options: [asymptotic, MC, hybrid] :param bool dropna: remove NaN values with True :param bool drop_underflow: do not take into account records in underflow bin when True (relevant when binning a numeric variable) :param bool drop_overflow: do not take into account records in overflow bin when True (relevant when binning a numeric variable) :param int njobs: number of parallel jobs used for simulation. default is -1. 1 uses no parallel jobs. :return: p-value, significance )r_rsr`r-ra)rSeriesrrrastyperrYr2rwhererrcrosstabrr{) r r5r_rsr-r`r}r~rrahist2ds r%rrsD  IaLL   ) ) 0 0 5 5 < IaLL   ) ) 0 0 5 5 < // IaLL   $ $ + IaLL   $ $ +  /(*AbhqDG|$$ %(*AbhqDG|$$ %  /(*AbhqDG|$$ %(*AbhqDG|$$ % [A   %F 6 " "vrv~ #/+     r')r)r\r]r^Nr;)r\r]r^rrr;)r]r^r\rrTTTr;) Nr]r^r\rrrTTTTr;) NrFr]r\rrr^TTTr;)r]rrr\r^TTTr;)5__doc__typingrrnumpyr2pandasrrQrrscipyrrrphikrrbinningr r statisticsr r r simulationr data_qualityrrutilsrrlistndarrayintfloatrJrLrXr[rrgrqr{rboolrdictrrrrr)r'r%rs  ########$$$$$$@@@@@@@@DDDDDD77777777------BBBBBBBB????????24&'&' rz! "&'+.&' 5% %&&'&'&'&'R RU Ru R R5 R R5 R R R R e5U5%<=P0 J# 5%<,#+ "0"0 "0 J"0 "0 "0  "0 "0 5%<"0"0"0"0P#+ >> > J> > >  > > 5%<>>>>F#+' 55 J5 55 5  5  5 5%<5555t$+'M!M!M! M!M!  M!  M!  M!M!M! M!\M!M!M!M!d#+'/199 999 9  9  9 T2:t+ ,9 9999 9\9999~/1#'+<< RZ "#< RZ "#< T2:t+ , <  <  < <<< <<< < 5%<<<<<D$'+>> RZ "#> RZ "#>> >  >  > >>> > 5%<>>>>>>r'