hMhH/VdZddlZddlZddlmZmZmZmZddlm Z ddl Z ddl ZddlmZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZdd lmZ dEdee e j!ej"fdefdZ# dFde j!de de de$de$de%de%de$d e$d!e$d"e&d#e j!d$e'd%e(d&e'd'e'd(e%d)df$d*Z)dd+d ddd,d-dd d.d/d0d1d2diiiifdej*d3e d4e'd5e'de$d6e%d7e%d8e'd9e'd:e$d;e$de$d?e'd@e+dAe+dBe+dCe+d)eej*ej*ee$ej*fee$e$fff(dDZ,dS)GayProject: PhiK - correlation analyzer library Created: 2018/09/06 Description: Functions to create nice correlation overview and matrix plots Authors: KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE. N)CallableDictTupleUnion)colors)PdfPages)bin_data)dq_check_nunique_values),outlier_significance_matrix_from_rebinned_df)global_phik_from_rebinned_dfphik_from_rebinned_df)significance_from_rebinned_df)guess_interval_colsFdatafuncc &|ddg}tj|sc|satj|} tj|} t t |dz dz} tj| | | }nt|tkst|tkrdtj|} |r|d} tj|} | r|d} t |dz} tj| | | }tj |||d| |ddtj |ddz z} |d|dz }tj | t ||z|| g|Rzd|d |rtj ||rtj||rtj|t |ddkrtjdSdS) a Create a histogram of the provided data and overlay with a function. :param list data: data :param function func: function of the type f(x, a, b, c) where parameters a, b, c are optional :param list funcparams: parameter values to be given to the function, to be specified as [a, b, c] :param xbins: specify binning of histogram, either by giving the number of bins or a list of bin edges :param labels: labels of histogram and function to be used in the legend :param xlabel: figure xlabel :param ylabel: figure ylabel :param title: figure title :param xlimit: x limits figure :param alpha: alpha histogram :return: Nr2r r)binslabelalpha) linewidthr)npanyminmaxintlenlinspacetypefloatplthistdiffplotxlabelylabeltitlelegend)rr funcparamsxbinslabelsr)r*r+xlimitrxminxmaxxnbinsxvalsbws K/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/phik/report.pyplot_hist_and_funcr7!s 6~b 6%== 0 0vd||vd||SYY^a'(( D$// e  tE{{e33vd||  !9Dvd||  !9DUQY D$//HTVAYe<<<< #2#J*Q. .E qE!H BH s4yy2~U 8Z 8 8 88AVTUY 6  6  % 6!9~~  correlationrRdYlGnT matrix_colorsx_labelsy_labels pdf_file_namer+vminvmax color_mapx_labely_labeltopmatrix_numbersprint_both_numbersfigsizeusetexidentity_layoutfontsize_factorreturnc F !t|tjstd|jdt |ks+|jddzt |ks Jd|jdt |ks+|jddzt |ks Jd| |} d} nP| jdt |ks Jd| jdt |ks Jd |rItjd |D}|ddd }| tjd | D} tjd |tj | \}}tj ||}| ||dd|} fd!t ||jddzkr5| tjt |n7| tjt |dz|!fd|Ddd|zt ||jddzkr5|tjt |n7|tjt |dz|!fd|Ddd|zt |dkr2t |ddkrtjdddddt |dkr2t |ddkrtjddddd||d |z!|r||d"|z!| r|| d"|z!||| s| gn| |g}t/| jdD]}t/| jdD]}t1|||}|d#|zkp|d#|zkptj|}d}t5|D]\}}| r|dkrd#}n|dkrd$}t1|||}tj|rd%nd&|}|rdnd'}|||dz||zf|d(d(d|z)tj|rPt=|} tj| d*d+d,tj | dSdS)-aCreate and plot correlation matrix. Copied with permission from the eskapade package (pip install eskapade) :param matrix_colors: input correlation matrix :param list x_labels: Labels for histogram x-axis bins :param list y_labels: Labels for histogram y-axis bins :param str pdf_file_name: if set, will store the plot in a pdf file :param str title: if set, title of the plot :param float vmin: minimum value of color legend (default is -1) :param float vmax: maximum value of color legend (default is +1) :param str x_label: Label for histogram x-axis :param str y_label: Label for histogram y-axis :param str color_map: color map passed to matplotlib pcolormesh. (default is 'RdYlGn') :param int top: only print the top 20 characters of x-labels and y-labels. (default is 20) :param matrix_numbers: input matrix used for plotting numbers. (default it matrix_colors) :param identity_layout: Plot diagonal from right top to bottom left (True) or bottom left to top right (False) z#matrix_colors is not a numpy array.rr z8matrix_colors shape inconsistent with number of y-labelsz8matrix_colors shape inconsistent with number of x-labelsNFz9matrix_numbers shape inconsistent with number of y-labelsz9matrix_numbers shape inconsistent with number of x-labelsc$g|] }|dddSNr.0as r6 z+plot_correlation_matrix..s"!A!A!Aa!DDbD'!A!A!Ar8rc$g|] }|dddSrRrSrTs r6rWz+plot_correlation_matrix..s"&G&G&G1q2w&G&G&Gr8text)rL)rK)rCrDw)cmap edgecolorrnormct|ttfr+tj|rdnd|}t |}t|kr |dddz}|S)z Get tick.NaNz{0:.0f}Nz...) isinstancer$r risnanformatstrr!)labrHs r6tickz%plot_correlation_matrix..tickso cE3< ( ( D8C==C%%i.>.>s.C.CC#hh s88c>>crc(U"C r8?c&g|] }|SrSrSrUrerfs r6rWz+plot_correlation_matrix..!'''sc'''r8vertical )rotationfontsizec&g|] }|SrSrSris r6rWz+plot_correlation_matrix..rjr8 horizontalxboth)axiswhichbottomrH labelbottomy)rsrtleftrightrv)rn gffffff?g?r_z{0:.2f}kcenter)xycolorhorizontalalignmentverticalalignmentrnpdftightrc bbox_inches pad_inches)!rarndarray TypeErrorshaper!arrayr%rcsubplotsr Normalize pcolormesh set_xticksarangeset_xticklabels set_yticksset_yticklabels tick_params set_title set_xlabel set_ylabelcolorbarranger$rb enumeratercannotate tight_layoutrsavefigclose)"r?r@rArBr+rCrDrErFrGrHrIrJrKrLrMrNfigaxr]img numbers_setij point_color white_condy_offsetmmatrixpointrrpdf_filerfs" ` @r6plot_correlation_matrixres}J mRZ 0 0?=>>>   "c(mm 3 3A"c(mm333A 433   "c(mm 3 3A"c(mm333A 433&"#A&# + +    F   #A&# + +    F   I!A!A=!A!A!ABB DDbD>  %X&G&G&G&G&GHHNF6&!!!!l7+++GC  D 1 1 1D --I   C  8}} +A.222 biH ..//// biH ..4555''''h'''o%  8}} +A.222 biH ..//// biH ..4555''''h'''o% 8}}c(1+..!33       8}}c(1+..!33      LLo!5L666> g_(< ===> g_(< ===LL!3W8W>'* + +~+A.//  A a 0 344KsTz))3:-)8K((  H&{33   6%(Avv#&a#'fQil++!#%Mi6F6Fu6M6M)2s CX.(0&./1   4M** HUANNNN  r8rlrgzlog-likelihood multinominali asymptoticpoisson interval_colsquantile do_outlierssignificance_thresholdcorrelation_thresholdnoise_correctionstore_each_plotlambda_significancesimulation_method nsim_chi2significance_method CI_methodverboseplot_phik_matrix_kwsplot_global_phik_kwsplot_significance_matrix_kwsplot_outlier_significance_kwsc |t||}t||\}}t}d}| r*tj|}||rdndz }d}|rt |}t||||d\}}| r |dz}||d<t||}t|j |j d d d d d d| }| |t|j fi||r+tj|ddd tj| r |dz}||d<t#||\}}tdg|d d dd dd | } | |t|fi| |r+tj|ddd tj| r |dz}||d<t%|| | | | }!t|!j |!j ddddd d| }"|" |t|!d j fi|"|r+tj|ddd tji}#|rtt)t+j|j dD]K\}$}%|%\}&}'t/|!j|&|'f|ks|j|&|'f|kr?t3||&|'g|| }(d!|%d"d#})|(j }*|(j }+|(j j},|(j j}-| r|d$|)z}|||)<t|*|+|,|-ddd%dd&|' }.|. |t|(j fi|.|(|#|)<|r+tj|ddd tjM|r,||d(<tj||||!|#|fS))a Create a correlation report for the given dataset. The following quantities are calculated: * The phik correlation matrix * The significance matrix * The outlier significances measured in pairs of variables. (optional) :param data: input dataframe :param interval_cols: list of columns names of columns containing interval data :param bins: number of bins, or a list of bin edges (same for all columns), or a dictionary where per column the bins are specified. (default=10) E.g.: bins = {'mileage':5, 'driver_age':[18,25,35,45,55,65,125]} :param quantile: when bins is an integer, uniform bins (False) or bins based on quantiles (True) :param do_outliers: Evaluate outlier significances of variable pairs (when True) :param pdf_file_name: file name of the pdf where the results are stored :param store_each_plot: store each plot in folder derived from pdf_file_name. If true, single pdf is no longer stored. Default is false. :param significance_threshold: evaluate outlier significance for all variable pairs with a significance of uncorrelation higher than this threshold :param correlation_threshold: evaluate outlier significance for all variable pairs with a phik correlation higher than this threshold :param noise_correction: Apply noise correction in phik calculation :param lambda_significance: test statistic used in significance calculation. Options: [pearson, log-likelihood] :param simulation_method: sampling method using in significance calculation. Options: [mutlinominal, row_product_multinominal, col_product_multinominal, hypergeometric] :param nsim_chi2: number of simulated datasets in significance calculation. :param significance_method: method for significance calculation. Options: [asymptotic, MC, hybrid] :param CI_method: method for uncertainty calculation for outlier significance calculation. Options: [poisson, exact_poisson] :param bool verbose: if False, do not print all interval columns that are guessed :param dict plot_phik_matrix_kws: kwargs passed to plot_correlation_matrix() to plot the phik matrix. updates the default plotting values. :param dict plot_global_phik_kws: kwargs passed to plot_correlation_matrix() to plot the global-phik vector. updates the default plotting values. :param dict plot_significance_matrix_kws: kwargs passed to plot_correlation_matrix() to plot significance matrix. updates the default plotting values. :param dict plot_outlier_significance_kws: kwargs passed to plot_correlation_matrix() to plot the outlier significances. updates the default plotting values. :returns: phik_matrix (pd.DataFrame), global_phik (np.array), significance_matrix (pd.DataFrame), outliers_overview (dictionary), output_files (dictionary) Nr/z./T)rrretbinszphik_matrix.pdf phik_matrixrr Blueszcorrelation $\phi_K$g?)r=g@) r@rArCrDrEr+rNrKrBrrrzglobal_phik.pdf global_phik)g @z$g_k$) r@rArCrDrKrEr+rNrBzsignificance_matrix.pdfsignificance_matrixr> significanceF) r@rArCrDr+rLrNrKrBr)r: _zpulls_{0:s}.pdfzoutlier significanceg333333?) r@rArFrGrCrDr+rMrNrBall) rr dictospathdirnamerr rcolumnsindexupdatervaluesr%rshowr rfillnar itertools combinationsabslocr copyjoinreplacenamercr)/rrrrrrBrrrrrrrrrrrrrr data_cleaninterval_cols_clean output_filesplot_file_namefolderr data_binned binning_dictrdefault_plot_phik_matrixr global_labelsdefault_plot_global_phikr default_plot_significance_matrixoutliers_overviewrcombc0c1 zvalues_dfcombixlabelsylabelsr)r*!default_plot_outlier_significances/ r6correlation_reportrs;@+D':: &=dM&R&R#J#66LN//)##T) +M** ('dXt!!!K 5"33&4 ]#' 5EFFK#$"  %$   ##$8999K.KK2JKKK HUANNNN  5"33&4 ]#!=%""K $  $   ##$8999KDD+CDDD HUANNNN  =";;.< *+7 (,$,$*  $ ( ( ($%++,HIII""1%%,0P HUANNNN  , !7 8KQ!O!OPP+ + GAtFB'+BF3447MMM?2r6*-BBBERH%**,,liJHHTNN**344E (G &G',F%*F 5!'*;*B*B5*I*I!I&4 U#04  , % #, 1 1 1 - . 4 45R S S S #!  %F   (2 e $  HUTUVVVV + U    r8)FNrrrNr )rr9rr r:rrr;NTr<FTr )-__doc__rrtypingrrrrmatplotlib.pyplotpyplotr%numpyrpandaspd matplotlibrmatplotlib.backends.backend_pdfrbinningr data_qualityr outliersr phikr rrrutilsrlistrSeriesr7rdr$r booltupler DataFramerrrSr8r6r s   ////////////444444111111BBBBBBEEEEEEEE777777&&&&&&    AA bj")+ ,A AAAAP!%# #ll:lll l  l  l llll lJlll l !l"#l$ %llllb $%#&!!/++!#!#)+*,)aa ,aa a  a  a"a!aaaaaaaa !a"#a$%a&#''a($()a* 2<tC,='>S#X NO+aaaaaar8