M/PhF^dZddlmZddlZddlmZGddZGddZdS) zL Created on Sat May 19 15:53:21 2018 Author: Josef Perktold License: BSD-3 ) defaultdictN SCADSmoothedceZdZdZdZdS)ScreeningResultsaResults for Variable Screening Note: Indices except for exog_idx and in the iterated case also idx_nonzero_batches are based on the combined [exog_keep, exog] array. Attributes ---------- results_final : instance Results instance returned by the final fit of the penalized model, i.e. after trimming exog with params below trimming threshold. results_pen : results instance Results instance of the penalized model before trimming. This includes variables from the last forward selection idx_nonzero index of exog columns in the final selection including exog_keep idx_exog index of exog columns in the final selection for exog candidates, i.e. without exog_keep idx_excl idx of excluded exog based on combined [exog_keep, exog] array. This is the complement of idx_nonzero converged : bool True if the iteration has converged and stopped before maxiter has been reached. False if maxiter has been reached. iterations : int number of iterations in the screening process. Each iteration consists of a forward selection step and a trimming step. history : dict of lists results collected for each iteration during the screening process 'idx_nonzero' 'params_keep'].append(start_params) history['idx_added'].append(idx) The ScreeningResults returned by `screen_exog_iterator` has additional attributes: idx_nonzero_batches : ndarray 2-D Two-dimensional array with batch index in the first column and variable index withing batch in the second column. They can be used jointly as index for the data in the exog_iterator. exog_final_names : list[str] 'var_' where `bidx` is the batch index and `idx` is the index of the selected column withing batch `bidx`. history_batches : dict of lists This provides information about the selected variables within each batch during the first round screening 'idx_nonzero' is based ond the array that includes exog_keep, while 'idx_exog' is the index based on the exog of the batch. c 8||_|jjdi|dS)N)screener__dict__update)selfr kwdss [/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/base/_screening.py__init__zScreeningResults.__init__?s*   $$t$$$$$N)__name__ __module__ __qualname____doc__rr rrrrs.//`%%%%%rrcBeZdZdZ ddZdd Zdd Z ddZdZdS)VariableScreeningaV Ultra-high, conditional sure independence screening This is an adjusted version of Fan's sure independence screening. Parameters ---------- model : instance of penalizing model examples: GLMPenalized, PoissonPenalized and LogitPenalized. The attributes of the model instance `pen_weight` and `penal` will be ignored. pen_weight : None or float penalization weight use in SCAD penalized MLE k_add : int number of exog to add during expansion or forward selection see Notes section for tie handling k_max_add : int maximum number of variables to include during variable addition, i.e. forward selection. default is 30 threshold_trim : float threshold for trimming parameters to zero, default is 1e-4 k_max_included : int maximum total number of variables to include in model. ranking_attr : str This determines the result attribute or model method that is used for the ranking of exog to include. The availability of attributes depends on the model. Default is 'resid_pearson', 'model.score_factor' can be used in GLM. ranking_project : bool If ranking_project is True, then the exog candidates for inclusion are first projected on the already included exog before the computation of the ranking measure. This brings the ranking measure closer to the statistic of a score test for variable addition. Notes ----- Status: experimental, tested only on a limited set of models and with a limited set of model options. Tie handling: If there are ties at the decision threshold, then all those tied exog columns are treated in the same way. During forward selection all exog columns with the same boundary value are included. During elimination, the tied columns are not dropped. Consequently, if ties are present, then the number of included exog can be larger than specified by k_add, k_max_add and k_max_included. The screening algorithm works similar to step wise regression. Each iteration of the screening algorithm includes a forward selection step where variables are added to the model, and a backwards selection step where variables are removed. In contrast to step wise regression, we add a fixed number of variables at each forward selection step. The backwards selection step is based on SCAD penalized estimation and trimming of variables with estimated coefficients below a threshold. The tuning parameters can be used to adjust the number of variables to add and to include depending on the size of the dataset. There is currently no automatic tuning parameter selection. Candidate explanatory variables should be standardized or should be on a similar scale because penalization and trimming are based on the absolute values of the parameters. TODOs and current limitations: freq_weights are not supported in this. Candidate ranking uses moment condition with resid_pearson or others without freq_weights. pearson_resid: GLM resid_pearson does not include freq_weights. variable names: do we keep track of those? currently made-up names currently only supports numpy arrays, no exog type check or conversion currently only single columns are selected, no terms (multi column exog) NT-C6? resid_pearsonc  ||_|j|_||_|jdd|jdd|j|_|j|_|jj d|_ t|j|_ | |_|||_n|j dz|_||_||_||_||_||_||_| |_dS)N pen_weightpenal )model __class__ model_class_get_init_kwds init_kwdspopendogexog exog_keepshapek_keeplennobs _get_penalrr use_weightsk_add k_max_addthreshold_trimk_max_included ranking_attrranking_project) r r!rr/r0r1r2r3r4r5s rrzVariableScreening.__init__s  ?--// <... 7D)))[ j&q)  OO __&&  !(DOO"i"nDO' ",,(.rc&tdd|S)z$create new Penalty instance g?r)c0weightsr)r r8s rr.zVariableScreening._get_penalsCFG<<<>Lvl..t4455q8HrdbfgsFc  |j}||j}|j}|j} |} |jd} t j|| f} | j\} }||ni}ddd}||tt}t j | t}t j | t j }t j | |}|||fi|j}d|_|jdi|}|j}d}g}t%|D]9}| dd|f} ||| |}t)|t)|ksJt j|ddd }t-|j| |jzt)|f}||}t j||||kf}t jt)|}||dt)|<|jr4t j t)|} d| d| <| |j_||| dd|ff|j|jd |j}|jd||dd d |}t j|j|jk}| |j!krit jt j|j|j! }!t j|j|!k}"t j"||"}d |d| <||}|rtG|tG|t)|} |j|}t j |tH}#d|#|<t j%|#d}|d &||d&||d&||d&|t)|t)|kr||k'rd }n|};t j'|d| t j | ksJ|jr?t j t)|} d| d| <|(| }$n|j}$||| dd|ff|$|jd |j}%|%jd||dd|}&d|D}'|'| d|&j)j*| d<tW|||&||| d| z ||||dz }(|(S)ascreen and select variables (columns) in exog Parameters ---------- exog : ndarray candidate explanatory variables that are screened for inclusion in the model endog : ndarray (optional) use a new endog in the screening model. This is not tested yet, and might not work correctly maxiter : int number of screening iterations method : str optimization method to use in fit, needs to be only of the gradient optimizers disp : bool display option for fit during optimization Returns ------- res_screen : instance of ScreeningResults The attribute `results_final` contains is the results instance with the final model selection. `idx_nonzero` contains the index of the selected exog in the full exog, combined exog that are always kept plust exog_candidates. see ScreeningResults for a full description NrF)maxiterdisp)dtyper)rK)rrT)method start_paramswarn_convergence skip_hessian idx_nonzerorK params_keep idx_added)r8)r[r\r]cg|]}d|zS)zvar%4dr ).0iis r z1VariableScreening.screen_exog..ds666B(R-666r) results_pen results_finalr_idx_exogidx_exclhistory converged iterationsr ),r#r'r)r+r*r? column_stackr rlistarangeintonesbool_r%rfitrBrangerRr,sortminr1r0 concatenatezerosr/rr8rIr2sumr3 logical_andprintboolnonzeroappendallr.r! exog_namesr))r r(r'rWr[rXfit_kwdsr#x0r+x1 k_currentxr-k_varsfkwdsrjr_rKrimod_penrJr\rkidx_olditrQmcsidx_thr thresholdidx start_params2r8 thresh_paramskeep2 mask_exclr mod_final res_finalxnamesress) r screen_exogzVariableScreening.screen_exogs:& =JE ^ HQK  ORH % %w f$0b"E22d##ic222 wvrx((9VV,,+eR::4>::'+))))~  ..? "? "B111h;B++GRd+CCHx==CMM1111'(##DDbD)C4>9tz+A3s88LMMGG I.+x98L/M!NOOCHSXX..M0(> ? ?9=9L8L!N w~..>~dE22!D&Md)K #d k"""K((I">$/Ld333I%*Ik "z),,Q/H M " ) )+ 6 6 6 FO " "4 ( ( ( M " ) ), 7 7 7 K ' ' , , ,K  CLL00 G+00221 !GGvk'6'*bi.?.??@@@@@   gc+..//G GGVG OOGO44EEJEKqK'82&++/?22#'.22 "IM./;38..%-.. 76+666.4VWWo "677+t-4/8-8*5fgg*>*G*2)0+4,.F ! ! ! rc|j}g}g}g}|D]}||d}||j||dd|j|d|z f||j|d|z t j|}||d}dt |D} dt |D} |j|d|z } t j| | } t j| | |_| |_ ||d} | |_ |S)ay batched version of screen exog This screens variables in a two step process: In the first step screen_exog is used on each element of the exog_iterator, and the batch winners are collected. In the second step all batch winners are combined into a new array of exog candidates and `screen_exog` is used to select a final model. Parameters ---------- exog_iterator : iterator over ndarrays Returns ------- res_screen_final : instance of ScreeningResults This is the instance returned by the second round call to `screen_exog`. Additional attributes are added to provide more information about the batched selection process. The index of final nonzero variables is `idx_nonzero_batches` which is a 2-dimensional array with batch index in the first column and variable index within batch in the second column. They can be used jointly as index for the data in the exog_iterator. see ScreeningResults for a full description r)rWNc,g|]\}}|D] }d||fz S)zvar%d_%dr rcbidxbatchrs rrez:VariableScreening.screen_exog_iterator..sJ///!,u(-//!$(4+5////rc&g|]\}}|D]}||fSr r rs rrez:VariableScreening.screen_exog_iterator..sE&&&#e$&&3K&&&&r)r_rh) r+rr~r_r?rm enumeratearrayidx_nonzero_batchesexog_final_nameshistory_batches)r exog_iteratorr+res_idx exog_winnerexog_idxex res_screenres_screen_finalexog_winner_namesidx_full ex_final_idx final_namesrjs rscreen_exog_iteratorz&VariableScreening.screen_exog_iteratorss<  F FB))"b)99J NN:1 2 2 2   r!!!Z%;FGG%Dv%M"MN O O O OOJ2677;fD E E E Eok22 ++K+DD//09(0C0C///&&'0':':&&&(3FGGrs$#####4444443%3%3%3%3%3%3%3%lk k k k k k k k k k r