M/Ph=6rdZddlmZddlZddlmZddlmZddlZ ddl Z efdZ ddddefd Z d d Z dd ZdS)aAnalyze a set of multiple variables with a linear models multiOLS: take a model and test it on a series of variables defined over a pandas dataset, returning a summary for each variable multigroup: take a boolean vector and the definition of several groups of variables and test if the group has a fraction of true values higher than the rest. It allows to test if the variables in the group are significantly more significant than outside the group. )dmatrixN)OLS)statsc |||fi|}tj|j|jd}tj|j|j|j|d}tjd|j id|j id}tj ||g }| S)zreturn a series containing the summary of a linear model All the exceding parameters will be redirected to the linear model )r2adj_r2)paramspvalsstd statistics_f_test)r r )fitpdSeriesrsquared rsquared_adj DataFramer pvaluesbsefvaluef_pvalueconcatunstackdropna) model_endog model_exog model_typekwargs model_resultr result_df fisher_df res_seriess _/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/sandbox/multilinear.py_model2dataframer$s :k:@@@@DDFFL,"7(57788J (;'3';%1%5,68899I  L4G(H(1<3H'IKKLLIIy122::<>> import statsmodels.api as sm >>> data = sm.datasets.longley.load_pandas() >>> df = data.exog >>> df['TOTEMP'] = data.endog This will perform the specified linear model on all the other columns of the dataframe >>> multiOLS('GNP + 1', df) This select only a certain subset of the columns >>> multiOLS('GNP + 0', df, ['GNPDEFL', 'TOTEMP', 'POP']) It is possible to specify a trasformation also on the target column, conforming to the patsy formula specification >>> multiOLS('GNP + 0', df, ['I(GNPDEFL**2)', 'center(TOTEMP)']) It is possible to specify the subset of the dataframe on which perform the analysis >> multiOLS('GNP + 1', df, subset=df.GNPDEFL > 90) Even a single column name can be given without enclosing it in a list >>> multiOLS('GNP + 0', df, 'GNPDEFL') NcHg|]}|jtk|v|S)dtypeobject).0name dataframemodels r# zmultiOLS..sAPPP"4.&88T=N=N=N=N=Nr%r/)data return_typez + 0)r2)r r zendogenous varsr methodalphaadj_)columns isinstancestrlocrKeyErrorr$rrT sort_valuesindexr.r multipletests)r0r/ column_listr5r6subsetrr col_resultsrcol_namerressummarysmtkey1key2p_values correcteds`` r#multiOLSrM+sLPPPPP (9PPP +s##$"m  M&) KYKHHHJ $ $ E#H-KK E E E!(V"3)DDDKKK E{J MMfMM # Hl;''Gi##%9$:;;G*GM  C11 t 7?? 4:&Cu===a@ (1 t#$$ NsA""BBTc>ddlm}m}dtz}dt jz}fd|D}fd|D} | r*d} t j| || dt|z} dtfd|Dz} d| | z z} d|| z z}d||z | z z}|r|n|}||g| | gg}|t j |d}| | ||f}t j || z|| zz }|||fS) ztest if the objects in the group are different from the general set. The test is performed on the pvalues set (ad a pandas series) over the group specified via a fisher exact test. r) fisher_exactchi2_contingencyg?c&g|] }|jv |Sr*r@r-crs r#r1z_test_group..s%:::qGM'9'91'9'9'9r%c&g|] }|jv |Sr*rRrSs r#r1z_test_group..s%:::Q1GM#9#9q#9#9#9r%zthe test is not well defined if the group has elements not presents in the significativity array. group name: {}, missing elements: {}c$g|] }| | Sr*r*rSs r#r1z_test_group..s"AAA!gajAAAAAr%r7) scipy.statsrOrPlennpsumloggingwarningformatarraylog)r group_namegroupexactrOrPtotalstotal_significant cross_indexmissings group_total group_sign group_nonsign extern_signextern_nonsigntesttablepvaluepartincreases` r# _test_grouprrs ;::::::: 3w<< FbfWoo-::::e:::K::::%:::G7; W55666K(((KsAAAA{AAABBBJ;34M*Z78KF%66FGN 6<<&6Dk *]J,G HE T"(5// " "1 %F }k> ADvv **[8:;;H 8T !!r%c"tj|}t|ddhkst dt |jdr|jjst diiiiiid}|D]\}}t||||}|d|d|<|d |d |<|d d|d |<|d d |d |<|d d |d|<|d d|d|<tj | d} |s | | j } tj} | | dd|d } | | d<| S)aC Test if the given groups are different from the total partition. Given a boolean array test if each group has a proportion of positives different than the complexive proportion. The test can be done as an exact Fisher test or approximated as a Chi squared test for more speed. Parameters ---------- pvals : pandas series of boolean the significativity of the variables under analysis groups : dict of list the name of each category of variables under exam. each one is a list of the variables included exact : bool, optional If True (default) use the fisher exact test, otherwise use the chi squared test for contingencies tables. For high number of elements in the array the fisher test can be significantly slower than the chi squared. keep_all : bool, optional if False it will drop those groups where the fraction of positive is below the expected result. If True (default) it will keep all the significant results. alpha : float, optional the significativity level for the pvalue correction on the whole set of groups (not inside the groups themselves). Returns ------- result_df: pandas dataframe for each group returns: pvals - the fisher p value of the test adj_pvals - the adjusted pvals increase - the log of the odd ratio between the internal significant ratio versus the external one _in_sign - significative elements inside the group _in_non - non significative elements inside the group _out_sign - significative elements outside the group _out_non - non significative elements outside the group Notes ----- This test allow to see if a category of variables is generally better suited to be described for the model. For example to see if a predictor gives more information on demographic or economical parameters, by creating two groups containing the endogenous variables of each category. This function is conceived for medical dataset with a lot of variables that can be easily grouped into functional groups. This is because The significativity of a group require a rather large number of composing elements. Examples -------- A toy example on a real dataset, the Guerry dataset from R >>> url = "https://raw.githubusercontent.com/vincentarelbundock/" >>> url = url + "Rdatasets/csv/HistData/Guerry.csv" >>> df = pd.read_csv(url, index_col='dept') evaluate the relationship between the various paramenters whith the Wealth >>> pvals = multiOLS('Wealth', df)['adj_pvals', '_f_test'] define the groups >>> groups = {} >>> groups['crime'] = ['Crime_prop', 'Infanticide', ... 'Crime_parents', 'Desertion', 'Crime_pers'] >>> groups['religion'] = ['Donation_clergy', 'Clergy', 'Donations'] >>> groups['wealth'] = ['Commerce', 'Lottery', 'Instruction', 'Literacy'] do the analysis of the significativity >>> multigroup(pvals < 0.05, groups) FTzthe series should be binary is_uniquez,series with duplicated index is not accepted)r rq_in_sign_in_non _out_sign_out_nonrr r7rqrurvrwrxr&r4 adj_pvals)rrsetunique ValueErrorhasattrr@rtitemsrrrr?rqrrA) r groupsrbkeep_allr6resultsr` group_listrFr rHrLs r# multigrouprsV Ie  E    E4= 0 06777u{K((I1FIGHHH G #),,..44 J%Z??'*1v$*-a& J'*-a&) J'),Q :&+.q6!9 Z(*-a&) J'' W%%11'::I 2i01  CIg&xuEEEaHI&Ik r%)T)TTr')__doc__patsyrpandasrstatsmodels.apirrnumpyrYr[r$rMrrrr*r%r#rs  !!!!!!9<.,0MMMM`!"!"!"!"Hddddddr%