hMh+dZddlZddlmZmZmZmZddlZddl Z ddl m Z ddlmZddlmZmZ d&deejee jfd ed ed ejfd Zdeejefd eejefd eejeffdZ d'de jdeeejefdeeeejefd edef dZ d(de jdeedeeeejefd ededed e jfdZdeeeee fd e jfdZ! d)de jdeded ed e jf d!Z" d*de jdeeeejfdeee eejefd ededed ededed ee jee jefffd"Z#d#ee jeejfd$e jeejgd ee jee jefffd%Z$dS)+aProject: PhiK - correlation analyzer library Created: 2018/09/06 Description: A set of rebinning functions, to help rebin two lists into a 2d histogram. Authors: KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE. N)ListOptionalTupleUnion) definitions)dq_check_nunique_values)array_like_to_dataframeguess_interval_colsFarrnbinsquantilereturnc f|rtjdd|dz}tj|tj||}|dxxt dt |dzt jjzcc<ntj|tj|}t dt |zt jj}tj||z tj|tj||dz}|S)a Create uniform or quantile bin-edges for the input array. :param arr: array like object with input data :param int nbins: the number of bin :param bool quantile: uniform bins (False) or bins based on quantiles (True) :returns: array with bin edges rg+=) nplinspacer isnanmaxabssys float_infomin)r r r quantilesxbins min_valueconstants L/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/phik/binning.py bin_edgesrs  K1eai00  C#/;; aCE!H -s~/ABBBF3 ~.// us9~~-s~/ABB  "&bhsmm^)<"="=uqy   Lrcftj||t}g}t j|j}tdt|D],}||vr&| ||dz ||f-tj |tj tj |<tj|tj |dk<tj|tj |t|k<||fS)z Index the data given the bin_edges. Underflow and overflow values are indicated. :param arr: array like object with input data :param bin_edges: list with bin edges. :returns: indexed data rr)r searchsortedastypeobjectpdSeries value_countsindexrangelenappendnanargwhererdefsUFOF)r r binned_arr bin_labels bin_indicesis r bin_arrayr44sC0077??JJ)J''4466 ? ? ?.0VJr{28C==))*04wJr{:?++,=AGJr{:Y7889 z !!r datacolsbinsretbinsc d}t|tr,|D](}||vr"td|)n#t|tt jfr|}|}i}|D]}t jt|t j s,t jt|t j r?t|| tt||}nt|trt jt||t j s2t jt||t j rDt|| tt|||}n^t||tt jfr||}n.|,tdt!t|zt#|| tj|\||<} |r| ||<|r||fS|S)aP Index the input DataFrame given the bin_edges for the columns specified in cols. :param DataFrame data: input data :param list cols: list of columns with numeric data which needs to be indexed :param bins: number of bins, or a list of bin edges (same for all columns), or a dictionary where per column the bins are specified. (default=10) E.g.: bins = {'mileage':5, 'driver_age':[18,25,35,45,55,65,125]} :param quantile: when bins is an integer, uniform bins (False) or bins based on quantiles (True) :returns: rebinned DataFrame :rtype: pandas.DataFrame Nz.column {0} is not included in bins dictionary.)r z1Unexpected type for bins. The found type was '%s') isinstancedict ValueErrorformatlistrndarraycopy issubdtypetypeintegerfloatingrr"floatintstrr4values) r7r8r9r r:rcol binned_data bins_dictr1s rbin_datarNWsP$ E$  C$ DKKCPP  D4, - -))++KI(( =dRZ 0 0 BM JJ 5 5  d3i..u55s4yy8TTTEE d # # }T$s)__bj99 "R]T#Y>> ""I$$U++Sc^^hDIbj'9:: "S  ]Cc$t**ooU (1c1A1A%1H1H1OQV'W'W$ C*  ('IcN&I%% rTdf interval_colsdropnaverbosec|t||}t|||\}}t||||d\}} || fS)a Index the input DataFrame with automatic bin_edges and interval columns. :param pd.DataFrame data_binned: input data :param list interval_cols: column names of columns with interval variables. :param bins: number of bins, or a list of bin edges (same for all columns), or a dictionary where per column the bins are specified. (default=10) E.g.: bins = {'mileage':5, 'driver_age':[18,25,35,45,55,65,125]} :param quantile: when bins is an integer, uniform bins (False) or bins based on quantiles (True) :param bool dropna: remove NaN values with True :param bool verbose: if False, do not print all interval columns that are guessed :return: phik correlation matrix N)rQT)r8r9r r:)r rrN) rOrPr9r rQrRdf_cleaninterval_cols_clean data_binned binning_dicts r auto_bin_datarXsq,+B88 %< M&%%%!H! !)*RV!!!K  $$rvalscg}|D]6\}}}||||g||||g7tj|gdddd}d|j_d|j_|S)a Create overview table of phik/significance data. :param list vals: list holding tuples of data for each variable pair formatted as ('var1', 'var2', value) :returns: symmetric table with phik/significances of all variable pairs :rtype: pandas.DataFrame )var1var2rY)columnsr[r\rY)r'r]rJN)r*r$ DataFrame pivot_tabler]namer')rYllc0c1v corr_matrixs r!create_correlation_overview_tablerfs B B 2r1+ 2r1+,r+C+C+CDDDPPfVQK $K!K rrVdrop_underflow drop_overflowc |j\}}|s!|tjd|r,|tjt jd|r,|tjt jd| ||g| d}|j |_|S)a Give binned 2d DataFrame of two columns of rebinned input DataFrame :param df: input data. DataFrame must contain exactly two columns :param bool dropna: remove NaN values with True :param bool drop_underflow: do not take into account records in underflow bin when True (relevant when binning a numeric variable) :param bool drop_overflow: do not take into account records in overflow bin when True (relevant when binning a numeric variable) :returns: histogram DataFrame T)inplacer)r]fillnar-NaNreplacer.rr+r/groupbycountto_frameunstack droplevel)rVrQrgrhrbrc df_datahists rhist2d_from_rebinned_dfrts$ FB 348T222;DGRVT:::;DGRVT::: RH%%b)//11::<<DDFFMMaPP&-7799K rc t|jdkrtd|t||}t ||d||\} } t | |||} |r| | fS| S)a Give binned 2d DataFrame of two columns of input DataFrame :param df: input data. DataFrame must contain exactly two columns :param interval_cols: columns with interval variables which need to be binned :param bins: number of bins, or a list of bin edges (same for all columns), or a dictionary where per column the bins are specified. (default=10) E.g.: bins = {'mileage':5, 'driver_age':[18,25,35,45,55,65,125]} :param bool quantile: when the number of bins is specified, use uniform binning (False) or quantile binning (True) :param bool dropna: remove NaN values with True :param bool drop_underflow: do not take into account records in underflow bin when True (relevant when binning a numeric variable) :param bool drop_overflow: do not take into account records in overflow bin when True (relevant when binning a numeric variable) :param bool verbose: if False, do not print all interval columns that are guessed :returns: histogram DataFrame z)DataFrame should contain only two columnsNT)r:r9r )rQrgrh)r)r]r>r rNrt) rOrPr9r rQrgrhr:rRrVrWdatahists rhist2drxs8 2:!DEEE+B88 ( M4dX!!!K'%# H&%% Orxyc <t||}t|fi|S)a Give binned 2d DataFrame of two input arrays :param x: input data. First array-like. :param y: input data. Second array-like. :param interval_cols: columns with interval variables which need to be binned :param bins: number of bins, or a list of bin edges (same for all columns), or a dictionary where per column the bins are specified. (default=10) E.g.: bins = {'mileage':5, 'driver_age':[18,25,35,45,55,65,125]} :param bool quantile: when the number of bins is specified, use uniform binning (False) or quantile binning (True) :param bool dropna: remove NaN values with True :param bool drop_underflow: do not take into account records in underflow bin when True (relevant when binning a numeric variable) :param bool drop_overflow: do not take into account records in overflow bin when True (relevant when binning a numeric variable) :returns: histogram DataFrame )r rx)ryrzkwargsrOs rhist2d_from_arrayr}(s)( !A & &B "    r)F)r5r6FF)Nr6FTT)TTT)Nr6FTTTFT)%__doc__rtypingrrrrnumpyrpandasr$phikrr-phik.data_qualityr phik.utilsr r rAr@r%rHboolrr4r^tupler=rNrXrIrGrfrtrxr}r5rrrs   ////////////$$$$$$555555CCCCCCCCKP rz4* +47CGZ4 " rz4  "-22:t3C-D " 2:t  " " " "J,./1 ;; ,; bj%' (; T2:t+ ,; ;  ;;;;@%)/1 "%"% "%D>"% T2:t+ ,"% "%  "%  "%\"%"%"%"%J uS#u_% &\4 !!! !! ! \ !!!!L8<68// /E$ "234/ UD"*d2 3/ /  /  //// 2<r|T12 23////d  RYbj () /1y$ .K  2<r|T12 23      r