hMh dZddlZddlZddlmZddlZddlZ d dej de de deej e ffd Z d ej de fd ZdS) a}Project: PhiK - correlation analyzer library Created: 2018/12/28 Description: A set of functions to check for data quality issues in input data. Authors: KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE. N)TupleTdf interval_colsdropnareturncfd|D}tttjt|z D]`}|dkr@t jd||ag|D]u}|dkrU|t jd||vtttjt|z D]}|dks |dkrW|rU|t jd|| }tj |}tdkr%| d fd |D}||fS) a Basic data quality checks per column in a DataFrame. The following checks are done: 1. For all non-interval variables, if the number of unique values per variable is larger than 100 a warning is printed. When the number of unique values is large, the variable is likely to be an interval variable. Calculation of phik will be slow(ish) for pairs of variables where one (or two) have many different values (i.e. many bins). 2. For all interval variables, the number of unique values must be at least two. If the number of unique values is zero (i.e. all NaN) the column is removed. If the number of unique values is one, it is not possible to automatically create a binning for this variable (as min and max are the same). The variable is therefore dropped, irrespective of whether dropna is True or False. 3. For all non-interval variables, the number of unique values must be at least either a) 1 if dropna=False (NaN is now also considered a valid category), or b) 2 if dropna=True The function returns a DataFrame where all columns with invalid data are removed. Also the list of interval_cols is updated and returned. :param pd.DataFrame df: input data :param list interval_cols: column names of columns with interval variables. :param bool dropna: remove NaN values when True :returns: cleaned data, updated list of interval columns c&g|] }|jv |S)columns).0colrs Q/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/phik/data_quality.py z+dq_check_nunique_values..6s%GGGSSBJ5F5FS5F5F5FzThe number of unique values of variable {0:s} is large: {1:d}. Are you sure this is not an interval variable? Analysis for pairs of variables including {0:s} can be slow.zSNot enough unique value for variable {0:s} for analysis {1:d}. Dropping this columnrT)r inplacecg|]}|v| Sr r )r r drop_colss rrz+dq_check_nunique_values..]s#TTTss)?S?Ss?S?S?Sr) sortedlistsetr nuniquewarningswarnformatappendcopylendrop)rrrr df_cleaninterval_cols_cleanrs` @rdq_check_nunique_valuesr$sV<HGGGMGGGMd3rz??S-?-??@@AA c7??  t # # MiioioC**jj   I c7??  q   S ! ! ! MellC**   d3rz??S-?-??@@AA c7??   ! !bgoo&7&71&<&<&<   S ! ! ! MellC**    wwyyH)M22 9~~ i 666TTTTmTTT ( ((rhist2dcd|jvs d|jvr@tjd|jd|jddS|jddkr2tjd|jd|jddkr2tjd|jddS) aIBasic data quality checks for a contingency table The Following checks are done: 1. There must be at least two bins in both the x and y direction. 2. If the number of bins in the x and/or y direction is larger than 100 a warning is printed. :param hist2d: contingency table :return: bool passed_check rrz9Too few unique values for variable x ({0:d}) or y ({1:d})FrzThe number of unique values of variable x is large: {0:d}. Are you sure this is not an interval variable? Analysis might be slow.zThe number of unique values of variable y is large: {0:d}. Are you sure this is not an interval variable? Analysis might be slow.T)shaperrr)r%s rdq_check_hist2dr(bs FLA-- G N N Qa     u |A  UU[U[ QVV   |A  UU[U[ QVV    4r)T)__doc__rrtypingrpandaspdnumpynp DataFramerboolr$ndarrayr(r rrr2s   ;?G)G) G)%)G)37G) 2< G)G)G)G)T#BJ#4######r