M/Ph dZddlZddlmZddlmZddlmZddlZ ddl m Z ddl m Z ddlmZdd lmZdd lmZ ddZddZdZGddZGddeZGddeZddZGddeZdS)zkInfluence and Outlier Measures Created on Sun Jan 29 11:16:09 2012 Author: Josef Perktold License: BSD-3 N)Appender)lzip) defaultdict)_plot_influence_doc)OLS) multipletests)cache_readonly)maybe_unwrap_resultsbonf皙?Fcddlm}|t|jjdd}t|dd}|+t |}t d|jjz|j } |rStj |  ddd} | | } |tj || }|jdz } |jtj | | d z} t#| || } tj| | | df}||dddf|k}||}nt'd}|2dd lm}||d d |dzgtj ||S|S)aU Outlier Tests for RegressionResults instances. Parameters ---------- model_results : RegressionResults Linear model results method : str - `bonferroni` : one-step correction - `sidak` : one-step correction - `holm-sidak` : - `holm` : - `simes-hochberg` : - `hommel` : - `fdr_bh` : Benjamini/Hochberg - `fdr_by` : Benjamini/Yekutieli See `statsmodels.stats.multitest.multipletests` for details. alpha : float familywise error rate labels : None or array_like If `labels` is not None, then it will be used as index to the returned pandas DataFrame. See also Returns below order : bool Whether or not to order the results by the absolute value of the studentized residuals. If labels are provided they will also be sorted. cutoff : None or float in [0, 1] If cutoff is not None, then the return only includes observations with multiple testing corrected p-values strictly below the cutoff. The returned array or dataframe can be empty if there are no outlier candidates at the specified cutoff. Returns ------- table : ndarray or DataFrame Returns either an ndarray or a DataFrame if labels is not None. Will attempt to get labels from model_results if available. The columns are the Studentized residuals, the unadjusted p-value, and the corrected p-value according to method. Notes ----- The unadjusted p-value is stats.t.sf(abs(resid), df) where df = df_resid - 1. rstatsN row_labels get_influencez=model_results object %s does not have a get_influence method.)alphamethod DataFrame student_residunadj_pz(p)columnsindex)scipyrgetattrmodeldatar AttributeError __class____name__resid_studentized_externalnpabsargsortasarraydf_residtsfrc_slicepandasr) model_resultsrrlabelsordercutoffrinflresultsresididxdfradj_pr!maskrs d/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/stats/outliers_influence.py outlier_testr<s\ ~,1<FF =/4 8 8D |&}55')0):)CDEE E DFF -E -fUmm##%%ddd+c   Z'',F  ! #Bgjj++a/G 'v > > >E 5q) *D AAArE{V#DzT{{ $$$$$$y"19fun!M!z&11$7999 9 Kc|dz}|jjjd}tj|j}|tj|dzz }tj||ddddf}tj |jj|f}|tj|dzdz}|jj |jj dzz }t|| }tj |dz |jd|} || S)arRamsey's RESET specification test for linear models This is a general specification test, for additional non-linear effects in a model. Parameters ---------- degree : int Maximum power to include in the RESET test. Powers 0 and 1 are excluded, so that degree tests powers 2, ..., degree of the fitted values. Notes ----- The test fits an auxiliary OLS regression where the design matrix, exog, is augmented by powers 2 to degree of the fitted values. Then it performs an F-test whether these additional terms are significant. If the p-value of the f-test is below a threshold, e.g. 0.1, then this indicates that there might be additional non-linear effects in the model and that the linear model is mis-specified. References ---------- https://en.wikipedia.org/wiki/Ramsey_RESET_test rrNr)r exogshaper&r) fittedvaluessqrtmeanvander column_stackendogrfiteyef_test) resdegreer2k_vars norm_valuesy_fitted_vanderrArHres_auxr_matrixs r; reset_ramseyrSls$6 QJE Y^ !! $F*S-..K)9(?(?(A(A B BBKi U33AAAssF;O ?CINO< = =DBGTQY$$Q'' ( ((D IOsy!399;; ;E%""$$Gvfqj$*Q-88H >>( # ##r=c|jd}tj|}|dd|f}tj||k}|dd|f}t ||j}dd|z z }|S)a Variance inflation factor, VIF, for one exogenous variable The variance inflation factor is a measure for the increase of the variance of the parameter estimates if an additional variable, given by exog_idx is added to the linear regression. It is a measure for multicollinearity of the design matrix, exog. One recommendation is that if VIF is greater than 5, then the explanatory variable given by exog_idx is highly collinear with the other explanatory variables, and the parameter estimates will have large standard errors because of this. Parameters ---------- exog : {ndarray, DataFrame} design matrix with all explanatory variables, as for example used in regression exog_idx : int index of the exogenous variable in the columns of exog Returns ------- float variance inflation factor Notes ----- This function does not save the auxiliary regression. See Also -------- xxx : class for regression diagnostics TODO: does not exist yet References ---------- https://en.wikipedia.org/wiki/Variance_inflation_factor rN?)rBr&r)arangerrIrsquared)rAexog_idxrNx_ir:x_noti r_squared_ivifs r;variance_inflation_factorr]sNZ]F :d  D qqq({ C 9V   (D !!!T']Fc6""&&((1K [ !C Jr=cneZdZdZeejd iddi d d Zdd Z dd Z dS)_BaseInfluenceMixinz=common methods between OLSInfluence and MLE/GLMInfluence extra_params_docNr cooks0?c |t|dod|jv}ddlm}|j||j|f||||||d|} n6t jd||j|f|||||j|j ||d|} | S)N_cacheres_looor)_influence_plot)externalr criterionsize plot_alphaaxz0Plot uses pearson residuals and exog hat matrix.)rirrjrkleverager6rlrm) hasattrrf$statsmodels.graphics.regressionplotsrhhat_matrix_diagr5warningswarnhat_matrix_exog_diagr6) selfrirrjrkrlrmkwargsrhrLs r;plot_influencez"_BaseInfluenceMixin.plot_influences  tX..L:3LHHHHHHH   +!/$,Jx(-,5D-7BJJCIJJCC ML M M M!/$,Jx(-,5D+/+D(, -7B JJ CI JJC  r=c ddlm}||\}}|d}t|j} t j| } |j| |fi||dkr t j| t j } nt j ||k} dt j| z} |j j j j} | t j| } |t j| d| t#| |t#| | d|}ddd }|j|fi||jd i||j|fi||S) Nr)utilsz Index Plotalllargeblack)fontsizecolor Observation)r)statsmodels.graphicsry create_mpl_axlenrHr&rVscatteronesbool_r'r5r r!r annotate_axeswherer set_ylabel set_xlabel set_title)ruyylabel thresholdtitlermkwdsryfignobsr large_pointspsizer1fonts r; _plot_indexz_BaseInfluenceMixin._plot_indexsx......%%b))R = E4: $ 5!$$t$$$   7422LL6!99y0LBGDMM!#(3 >Yt__F  ,!7!7!:F!%eQ!%ufe!4!4g!#%% 11 f%%%%% ,,t,,, U##d### r=c |}|d}|dkr,|jdd|f}d|jjj|z} n|dr|jd}d} n|ds|d r |j}d } n`|dr|jd}d} n;|d r |j}d } nt||}|||}|} |j || f|||d |} | S)a.index plot for influence attributes Parameters ---------- y_var : str Name of attribute or shortcut for predefined attributes that will be plotted on the y-axis. threshold : None or float Threshold for adding annotation with observation labels. Observations for which the absolute value of the y_var is larger than the threshold will be annotated. Set to a negative number to label all observations or to a large number to have no annotation. title : str If provided, the title will replace the default "Index Plot" title. ax : matplolib axis instance The plot will be added to the `ax` if provided, otherwise a new figure is created. idx : {None, int} Some attributes require an additional index to select the y-var. In dfbetas this refers to the column indes. kwds : optional keywords Keywords will be used in the call to matplotlib scatter function. Nrzdfbetaz DFBETA for cookrzCook's distancehatleverz!Leverage (diagonal of hat matrix) resid_stuz Internally Studentized Residuals)rrrm) dfbetasr5r exog_names startswithcooks_distancerqresid_studentizedrr) ruy_varrrrmr7rrjrrrs r; plot_indexz_BaseInfluenceMixin.plot_indexsY2  I   QQQV$A"T\%7%B3%GGFF  ! !& ) ) #A&A&FF  ! !% ( ( I,@,@,I,I $A8FF  ! !& ) ) #A&A&FF  ! !+ . . &A7FFe$$AcFFdq&.IU"$..(,.. r=)Nr rbrcrdN)NNN)rbNNNN) r$ __module__ __qualname____doc__rrformatrwrrrr=r;r_r_sX(!(DD,>+CDDEEAH37FE*<HL666666r=r_ceZdZdZ ddZedZedZedZedZ edZ ed Z ed Z d Z ddZedZedZedZdZdS) MLEInfluencea Global Influence and outlier measures (experimental) Parameters ---------- results : instance of results class This only works for model and results classes that have the necessary helper methods. other arguments : Those are only available to override default behavior and are used instead of the corresponding attribute of the results class. By default resid_pearson is used as resid. Attributes ---------- hat_matrix_diag (hii) : This is the generalized leverage computed as the local derivative of fittedvalues (predicted mean) with respect to the observed response for each observation. Not available for ZeroInflated models because of nondifferentiability. d_params : Change in parameters computed with one Newton step using the full Hessian corrected by division by (1 - hii). If hat_matrix_diag is not available, then the division by (1 - hii) is not included. dbetas : change in parameters divided by the standard error of parameters from the full model results, ``bse``. cooks_distance : quadratic form for change in parameters weighted by ``cov_params`` from the full model divided by the number of variables. It includes p-values based on the F-distribution which are only approximate outside of linear Gaussian models. resid_studentized : In the general MLE case resid_studentized are computed from the score residuals scaled by hessian factor and leverage. This does not use ``cov_params``. d_fittedvalues : local change of expected mean given the change in the parameters as computed in ``d_params``. d_fittedvalues_scaled : same as d_fittedvalues but scaled by the standard errors of a predicted mean of the response. params_one : is the one step parameter estimate computed as ``params`` from the full sample minus ``d_params``. Notes ----- MLEInfluence uses generic definitions based on maximum likelihood models. MLEInfluence produces the same results as GLMInfluence for canonical links (verified for GLM Binomial, Poisson and Gaussian). There will be some differences for non-canonical links or if a robust cov_type is used. For example, the generalized leverage differs from the definition of the GLM hat matrix in the case of Probit, which corresponds to family Binomial with a non-canonical link. The extension to non-standard models, e.g. multi-link model like BetaModel and the ZeroInflated models is still experimental and might still change. Additonally, ZeroInflated and some threshold models have a nondifferentiability in the generalized leverage. How this case is treated might also change. Warning: This does currently not work for constrained or penalized models, e.g. models estimated with fit_constrained or fit_regularized. This has not yet been tested for correctness when offset or exposure are used, although they should be supported by the code. status: experimental, This class will need changes to support different kinds of models, e.g. extra parameters in discrete.NegativeBinomial or two-part models like ZeroInflatedPoisson. Nct|x|_}|jjj\|_|_tj|j |_ ||n |jj |_ ||n |jj|_||n|j |_ |||_ nCt|dd|_ |j &|j tj|j z |_ ||n||_|jj|_|jj|jj |_|jj|jj |_| ||_dSdS)N resid_pearson)r r5r rArBrrNr&rkparamsk_paramsrHscaler6rrD cov_paramsr# model_classhessian score_obs_hat_matrix_diag)rur5r6rHrArqrrs r;__init__zMLEInfluence.__init__sH"6g!>!>> w!(!3!9 4;// #/UUW]5H  ,DD'-2D #/UUW]  DJJ /4@@DJz%!Z"'$**=*== )3)?:: ' 2 2 4 4 "=2|)11$,2EFF +55dl6IJJ  &$3D ! ! ! ' &r=ct|dr|jS |jj|jj}n#t $rd}YnwxYw|tjdtdS|jj |jj}|tj |j |jjzd}|S)zwDiagonal of the generalized leverage This is the analogue of the hat matrix diagonal for general MLE. rNz0hat matrix is not available, missing derivativesr)rorr5r _deriv_score_obs_dendogrNotImplementedErrorrrrs UserWarning_deriv_mean_dparamsr&linalgsolverTsum)rudsdydmu_dphs r;rqzMLEInfluence.hat_matrix_diags 4+ , , )( ( <%== #%%DD"   DDD  < ML% ' ' '4#77 8KLLbioot|mTV<<> > C CA F Fs)A AAct|jjdd}|tj|}n|j}|tj|jz dS)z>Diagonal of the hat_matrix using only exog as in OLS _get_exogsNr) rr5r r&rGrArpinvrr)ru get_exogsrAs r;rtz!MLEInfluence.hat_matrix_exog_diagsh DL. dCC  ?99;;//DD9Dry~~d++--221555r=c|jd|jz }tj|j|jj}|j|d|jz dddfz}|S)zApproximate change in parameter estimates when dropping observation. This uses one-step approximation of the parameter change to deleting one observation. rNr)rrr&rrrrrq)ruso_notibeta_is r;d_paramszMLEInfluence.d_paramssi.$$Q''$.8wy99;   + q4//D9 9F r=c.|j|jjz }|S)zScaled change in parameter estimates. The one-step change of parameters in d_params is rescaled by dividing by the standard error of the parameter estimate given by results.bse. )rr5bserurs r;rzMLEInfluence.dfbetass!11 r=c*|jj|jz S)zParameter estimate based on one-step approximation. This the one step parameter estimate computed as ``params`` from the full sample minus ``d_params``. )r5rrrus r; params_onezMLEInfluence.params_ones|"T]22r=c|jtj|j|jjjzd}||jz}ddlm }|j ||j|j j }||fS)aCook's distance and p-values. Based on one step approximation d_params and on results.cov_params Cook's distance divides by the number of explanatory variables. p-values are based on the F-distribution which are only approximate outside of linear Gaussian models. Warning: The definition of p-values might change if we switch to using chi-square distribution instead of F-distribution, or if we make it dependent on the fit keyword use_t. rrr)rr&rrrrrrrrfr,r5r*)rucooks_d2rpvalss r;rzMLEInfluence.cooks_distancesMBIOODO48MO%E%EEFGHKA DM!  8T]DL4IJJr=cJ|jtjd|jz z S)a@studentized default residuals. This uses the residual in `resid` attribute, which is by default resid_pearson and studentizes is using the generalized leverage. self.resid / np.sqrt(1 - self.hat_matrix_diag) Studentized residuals are not available if hat_matrix_diag is None. r)r6r&rDrqrs r;rzMLEInfluence.resid_studentizeds#zBGA(<$<====r=cddlm}|jj|jj}|jj|jj}t|tr|d}t|tr|d}t|jj|s| }|tj |z tj d|j z z S)aScore residual divided by sqrt of hessian factor. experimental, agrees with GLMInfluence for Binomial and Gaussian. This corresponds to considering the linear predictors as parameters of the model. Note: Nhis might have nan values if second derivative, hessian_factor, is positive, i.e. loglikelihood is not globally concave w.r.t. linear predictor. (This occured in an example for GeneralizedPoisson) r)GLMr) +statsmodels.genmod.generalized_linear_modelrr5r score_factorrhessian_factor isinstancetupler&rDrq)rurr,hfs r;resid_score_factorzMLEInfluence.resid_score_factor s DCCCCC \  , ,T\-@ A A \  . .t|/B C C b%  AB b%  AB$,,c22 BBGBKK"'!d.B*B"C"CCCr=TFcd|jj|jj}|jj|jj}| |dd|f}||dddf|f}|rB|jt j| |jz d}n*|t j t j | z }|rK|r |t j d|j z z}n)|t j d|j dddfz z}|S)aScore observations scaled by inverse hessian. Score residual in resid_score are defined in analogy to a score test statistic for each observation. Parameters ---------- joint : bool If joint is true, then a quadratic form similar to score_test is returned for each observation. If joint is false, then standardized score_obs are returned. The returned array is two-dimensional index : ndarray (optional) Optional index to select a subset of score_obs columns. By default, all columns of score_obs will be used. studentize : bool If studentize is true, the the scaled residuals are also studentized using the generalized leverage. Returns ------- array : 1-D or 2-D residuals Notes ----- Status: experimental Because of the one srep approacimation of d_params, score residuals are identical to cooks_distance, except for - cooks_distance is normalized by the number of parameters - cooks_distance uses cov_params, resid_score is based on Hessian. This will make them differ in the case of robust cov_params. Nrr) r5r rrrrr&rrrrDdiagrq)rujointr studentizerhessr6s r; resid_scorezMLEInfluence.resid_score#s(JL&001DEE |!))$,*=>>  !!!!U(+Iaaag-.D  8[29??D5)+#F#FFKKANNEE 7 77E  D DT%9!9:::T%9!!!T'%B!BCCC r=ctj5d}tjd|t|j}dddn #1swxYwY|S)Nz0linear keyword is deprecated, use which="linear"ignore)messagecategory)rrcatch_warningsfilterwarnings FutureWarningr5get_prediction)rumsgpreds r;_get_predictionzMLEInfluence._get_prediction\s $ & & 1 1DC  #Hc-: < < < <<..00D  1 1 1 1 1 1 1 1 1 1 1 1 1 1 1  s8AAActj|jj}|jj|}||jzdS)a2Change in expected response, fittedvalues. Local change of expected mean given the change in the parameters as computed in d_params. Notes ----- This uses the one-step approximation of the parameter change to deleting one observation ``d_params``. r)r&r)r5rr rrr)rurderivs r;d_fittedvalueszMLEInfluence.d_fittedvaluesgsKDL/00 "66v>> %**1---r=c*|j|jjz S)a Change in fittedvalues scaled by standard errors. This uses one-step approximation of the parameter change to deleting one observation ``d_params``, and divides by the standard errors for the predicted mean provided by results.get_prediction. )rrsers r;d_fittedvalues_scaledz"MLEInfluence.d_fittedvalues_scaledxs"T%9%<<z.MLEInfluence.summary_frame..777avz777r=N)cooks_dstandard_residhat_diagdffits_internalr)rrr)r/rr5r r!rxnamesrqdictrrrrjoinrurr!r beta_labels summary_datars r; summary_framezMLEInfluence.summary_frames . %$$$$$|!&_ 774;777    +$9T+A.#5- $ : &<&<&< ! """LL%9T+A.!% : &<&<&< ! """L4<!+---{{<(((r=)NNNNNN)TNF)r$rrrrr rqrtrrrrrrrrrpropertyrr rr=r;rr:sBBH>B>B44446^4 6 6^ 6  ^ ^33^3^2 > >^ >DDD07777r^..^.  = =X =2)2)2)2)2)r=rceZdZdZdZedZedZedZedZ edZ edZ ed Z ed Z d d Zed ZedZedZedZedZedZedZedZedZedZedZd!dZdZedZdZd"dZd S)# OLSInfluencea[class to calculate outlier and influence measures for OLS result Parameters ---------- results : RegressionResults currently assumes the results are from an OLS regression Notes ----- One part of the results can be calculated without any auxiliary regression (some of which have the `_internal` postfix in the name. Other statistics require leave-one-observation-out (LOOO) auxiliary regression, and will be slower (mainly results with `_external` postfix in the name). The auxiliary LOOO regression only the required results are stored. Using the LOO measures is currently only recommended if the data set is not too large. One possible approach for LOOO measures would be to identify possible problem observations with the _internal measures, and then run the leave-one-observation-out only with observations that are possible outliers. (However, this is not yet available in an automated way.) This should be extended to general least squares. The leave-one-variable-out (LOVO) auxiliary regression are currently not used. ct||_|jjj\|_|_|jj|_|jj|_|j|_|jj |_ |j |_ i|_ i|_dSN)r r5r rArBrrNrHr6r#r mse_residraux_regression_exogaux_regression_endog)rur5s r;rzOLSInfluence.__init__sw+G44 !(!3!9 4;]( M& ] "=2& #% $&!!!r=cd|j|jjjjzdS)zDiagonal of the hat_matrix for OLS Notes ----- temporarily calculated here, this should go to model class r)rAr5r pinv_wexogrrrs r;rqzOLSInfluence.hat_matrix_diags* DL.9;;@@CCCr=c*|j}|jd|z z S)zPRESS residuals rrqr6ruhiis r; resid_presszOLSInfluence.resid_presss"zQW%%r=c0|j}|j|zd|z z S)zInfluence measure matches the influence measure that gretl reports u * h / (1 - h) where u are the residuals and h is the diagonal of the hat_matrix rrrs r; influencezOLSInfluence.influences""zC1s7++r=c |j}|d|z z S)zFactor of diagonal of hat_matrix used in influence this might be useful for internal reuse h / (1 - h) r)rqrs r;hat_diag_factorzOLSInfluence.hat_diag_factors"a#gr=c@tj|j|jS)z0Error sum of squares of PRESS residuals )r&dotrrs r; ess_presszOLSInfluence.ess_press svd&(8999r=c|jS)zStudentized residuals using variance from OLS alias for resid_studentized_internal for compatibility with MLEInfluence this uses sigma from original estimate and does not require leave one out loop )resid_studentized_internalrs r;rzOLSInfluence.resid_studentizeds ..r=c.|dS)zStudentized residuals using variance from OLS this uses sigma from original estimate does not require leave one out loop Nsigma)get_resid_studentized_externalrs r;r"z'OLSInfluence.resid_studentized_internals222>>>r=c`tj|j}||S)zStudentized residuals using LOOO variance this uses sigma from leave-one-out estimates requires leave one out loop for observations r$)r&rDsigma2_not_obsir&)ru sigma_looos r;r%z'OLSInfluence.resid_studentized_external&s-WT122 222DDDr=Nc|j}||j}tj|}|j|z tjd|z z S)acalculate studentized residuals Parameters ---------- sigma : None or float estimate of the standard deviation of the residuals. If None, then the estimate from the regression results is used. Returns ------- stzd_resid : ndarray studentized residuals Notes ----- studentized residuals are defined as :: resid / sigma / np.sqrt(1 - hii) where resid are the residuals from the regression, sigma is an estimate of the standard deviation of the residuals, and hii is the diagonal of the hat_matrix. Nr)rqrr&rDr6)rur%r sigma2_ests r;r&z+OLSInfluence.get_resid_studentized_external1sG0" =JGJ''EzE!BGAG$4$444r=c|j}|jdz|jz }||d|z z z}ddlm}|j||j|jj}||fS)aa Cooks distance Uses original results, no nobs loop References ---------- .. [*] Eubank, R. L. (1999). Nonparametric regression and spline smoothing. CRC press. .. [*] Cook's distance. (n.d.). In Wikipedia. July 2019, from https://en.wikipedia.org/wiki/Cook%27s_distance rrrr rqrrNrrrr,r5r*rurrrrs r;rzOLSInfluence.cooks_distanceRsn")Q.<C1s7O# 8T[$,2GHHr=c|j}|jtj|d|z z z}dtj|jdz|jz z}||fS)zdffits measure for influence of an observation based on resid_studentized_internal uses original results, no nobs loop rrrU)rqr"r&rDrNrrurdffits_dffits_thresholds r;rzOLSInfluence.dffits_internalmsZ"1BGC1s7O4L4LLrwt{R'7$)'CDDD(((r=c|j}|jtj|d|z z z}dtj|jdz|jz z}||fS)a dffits measure for influence of an observation based on resid_studentized_external, uses results from leave-one-observation-out loop It is recommended that observations with dffits large than a threshold of 2 sqrt{k / n} where k is the number of parameters, should be investigated. Returns ------- dffits : float dffits_threshold : float References ---------- `Wikipedia `_ rrrU)rqr%r&rDrNrr0s r;dffitszOLSInfluence.dffits|sZ0"1BGC1s7O4L4LLrwt{R'7$)'CDDD(((r=c|jj|jz }|tj|jdddfz}|tjtj|jjz}|S)Jdfbetas uses results from leave-one-observation-out loop N)r5rparams_not_obsir&rDr(rnormalized_cov_params)rurs r;rzOLSInfluence.dfbetassb ,%(<<274/4899927274<#EFFGGGr=c.|jj|jz }|S)r6)r5rr7)rurs r;rzOLSInfluence.dfbetas $t';; r=c@tj|jdS)zerror variance for all LOOO regressions This is 'mse_resid' from each auxiliary regression. uses results from leave-one-observation-out loop rr&r) _res_looors r;r(zOLSInfluence.sigma2_not_obsisz$.5666r=c@tj|jdS)zoparameter estimates for all LOOO regressions uses results from leave-one-observation-out loop rr;rs r;r7zOLSInfluence.params_not_obsis z$.2333r=c@tj|jdS)ztdeterminant of cov_params of all LOOO regressions uses results from leave-one-observation-out loop det_cov_paramsr;rs r;det_cov_params_not_obsiz$OLSInfluence.det_cov_params_not_obsis z$.)9:;;;r=c|jtj|jz }|S)zcovariance ratio between LOOO and original This uses determinant of the estimate of the parameter covariance from leave-one-out estimates. requires leave one out loop for observations )r@r&rdetr5r)ru cov_ratios r;rCzOLSInfluence.cov_ratios71y}}T\%<%<%>%>??@ r=c&|jd|jz zS)zestimate of variance of the residuals :: sigma2 = sigma2_OLS * (1 - hii) where hii is the diagonal of the hat matrix r)rrqrs r; resid_varzOLSInfluence.resid_varszQ!5566r=c4tj|jS)zmestimate of standard deviation of the residuals See Also -------- resid_var )r&rDrErs r; resid_stdzOLSInfluence.resid_stdswt~&&&r=rHTc|dkr1|j}t||r||S|jjj}nE |j||n#t $rYnwxYw|j|}i}|jdd|f}|jjd}tj ||k}|jdd|f}t|| } |r| ||<| S)aregression results from LOVO auxiliary regression with cache The result instances are stored, which could use a large amount of memory if the datasets are large. There are too many combinations to store them all, except for small problems. Parameters ---------- drop_idx : int index of exog that is dropped from the regression endog_idx : 'endog' or int If 'endog', then the endogenous variable of the result instance is regressed on the exogenous variables, excluding the one at drop_idx. If endog_idx is an integer, then the exog with that index is regressed with OLS on all other exogenous variables. (The latter is the auxiliary regression for the variance inflation factor.) this needs more thought, memory versus speed not yet used in any other parts, not sufficiently tested rHNr) rror5r rHrKeyErrorrArBr&rVrrI) rudrop_idx endog_idxstorestoredrYrNr:rZrLs r; _ols_xnotizOLSInfluence._ols_xnotis2   .Fvx(( (h'',$*CC (3H===    -i8FF)AAAyL)C#y  H,111d7##v""$$  #"F8  sA AAcZddlm}|jjj}|j}||j}tt}|D]e\}}|D]]} | ||dd|f } ||  t| | ^f|S)az regress endog on exog without one of the variables This uses a k_vars loop, only attributes of the OLS instance are stored. Parameters ---------- attributes : list[str] These are the names of the attributes of the auxiliary OLS results instance that are stored and returned. not yet used r LeaveOneOutN) #statsmodels.sandbox.tools.cross_valrQr5r rHrArNrlistrrIappendr) ru attributesrQrHrAcv_iterres_looinidxoutidxattres_is r;_get_drop_varizOLSInfluence._get_drop_varis DCCCCC "(y+dk**d##$ 9 9ME6! 9 9((QQQX??CCEE ##GE3$7$78888 9r=cddlm}d}|jjj}|jjj}t j|jt}t j|jt}t j|jt}||j }|D][\} } | || ||  } | j || <| j|| <|| || <\t|||S)a9collect required results from the LOOO loop all results will be attached. currently only 'params', 'mse_resid', 'det_cov_params' are stored regresses endog on exog dropping one observation at a time this uses a nobs loop, only attributes of the OLS instance are stored. rrPcdtj|Srr&rrBrrLs r;get_det_cov_paramsz2OLSInfluence._res_looo..get_det_cov_paramsIs 9==!1!122 2r=dtype)rrr?)rRrQr5r rHrAr&zerosrBfloatrrrIrrr) rurQrarHrArrr?rVrXrYr[s r;r<zOLSInfluence._res_looo<s DCCCCC 3 3 3 "(|!&$*E222HU[666 %+U;;;+di(($ ? ?ME6$$U5\4;??CCEEE"\F6N %If %7%7%>%>N6 " "6Y#1333 3r=c Pddlm}|jjj}|j}d|jD}|t|jd|j |j |j d|j |j d|}||j||}||S)a Creates a DataFrame with all available influence results. Returns ------- frame : DataFrame A DataFrame with all results. Notes ----- The resultant DataFrame contains six variables in addition to the DFBETAS. These are: * cooks_d : Cook's Distance defined in `Influence.cooks_distance` * standard_resid : Standardized residuals defined in `Influence.resid_studentized_internal` * hat_diag : The diagonal of the projection, or hat, matrix defined in `Influence.hat_matrix_diag` * dffits_internal : DFFITS statistics using internally Studentized residuals defined in `Influence.dffits_internal` * dffits : DFFITS statistics using externally Studentized residuals defined in `Influence.dffits` * student_resid : Externally Studentized residuals defined in `Influence.resid_studentized_external` rrcg|]}d|zSrrrs r;rz.OLSInfluence.summary_frame..|rr=)rrrrrr4rr)r/rr5r r!rrrrr"rqrr%r4rrrs r;r zOLSInfluence.summary_frame]s4 %$$$$$|!&_ 774;777 !y'*:) 039;q> " " "  4<!+---{{<(((r=%6.3fc dtj|jfd|jfd|jjfd|jdfd|jfd|jfd|j dfd |j fd |j dfg }t|\}}tj |}||_dd lm}dd lm}m}dd lm}||} ||} dg|g|jddz zz| d<|||| | S)acreate a summary table with all influence and outlier measures This does currently not distinguish between statistics that can be calculated from the original regression results and for which a leave-one-observation-out loop is needed Returns ------- res : SimpleTable SimpleTable instance with the results, can be printed Notes ----- This also attaches table_data to the instance. obsrHz fitted valuezCook's drzstudent. residualzhat diagzdffits internalzext.stud. residualr4deepcopy SimpleTabledefault_html_fmtfmt_base%4dr data_fmtsheaderstxt_fmthtml_fmt)r&rVrrHr5rCrr"rqrr%r4rrG table_datacopyrlstatsmodels.iolib.tablernro!statsmodels.iolib.tableformattingrqrB) ru float_fmt table_rawcolnamesr!rlrnrorqfmtfmt_htmls r; summary_tablezOLSInfluence.summary_tablesK:RYty112tz*%t|'@A!4#6q#9:*D,KL $"67)4+?+BC+T-LM A/  y)$t$$!!!!!!IIIIIIII>>>>>>hx  8,--!7i[DJqMA4E%FFK{43$,... .r=r)rHT)rh) r$rrrrr rqrrrr rr"r%r&rrr4rrr(r r7r@rCrErGrNr\r<r rrr=r;r r s6 ' ' 'DD^D&&^& ,,^,^::^: //^/??^?EE^E5555B^4 ) )^ )))^)8^^77^744X4<<X<  ^  7 7^ 7''^'2222h:33^3@/)/)/)b3.3.3.3.3.3.r=r cddlm}ddlm}t |}t j|j|jz}|j |dz |j }t j |j ||zz |j ||zzg}|||}|\} } } t j | | f} t j|jd|jz z} t j t j|jdz|jj|j ||dddf|dddf| dddf| dddf|j| |j|jdg }|}gd}|}dd lm}dd lm}m}dd lm}||}||}d gd g|jddz zz|d<|||||}|||fS)a Generate summary table of outlier and influence similar to SAS Parameters ---------- alpha : float significance level for confidence interval Returns ------- st : SimpleTable table with results that can be printed data : ndarray calculated measures and statistics for the table ss2 : list[str] column_names for table (Note: rows of table are observations) rr)wls_prediction_stdg@)rrN) ObszDep Var PopulationzPredicted ValuezStd Error Mean PredictzMean ci 95% lowzMean ci 95% uppzPredict ci 95% lowzPredict ci 95% uppResidualzStd Error ResidualzStudent ResidualzCook's Drkrmrprrrhrsrt)rr&statsmodels.sandbox.regression.predstdrr r&rDrqrr+isfr*rGrCrVrr rHr6r"rryrlrzrnror{rqrB)rLrrrr4predict_mean_setppfpredict_mean_citmp predict_sepredict_ci_lowpredict_ci_upp predict_ciresid_setable_smr!ss2r~rlrnrorqrrsts r;rrsZ&IIIIII   Dgd2S]BCCO 7;;urz3< 0 0Do 4/11 4/11'344O  S . . .C14.J..!ABBJws}D,@(@ABBH #(a  11111a4111a4  ' A   H D D D DCHEEEEEEEE:::::: (8  Cx())Hw'djma.?!@@C  T8S& ( ( (B tS=r=ceZdZdZedZedZefdZedZe dZ e dZ e dZ e d Z ed ZxZS) GLMInfluencea3Influence and outlier measures (experimental) This uses partly formulas specific to GLM, specifically cooks_distance is based on the hessian, i.e. observed or expected information matrix and not on cov_params, in contrast to MLEInfluence. Standardization for changes in parameters, in fittedvalues and in the linear predictor are based on cov_params. Parameters ---------- results : instance of results class This only works for model and results classes that have the necessary helper methods. other arguments are only to override default behavior and are used instead of the corresponding attribute of the results class. By default resid_pearson is used as resid. Attributes ---------- dbetas change in parameters divided by the standard error of parameters from the full model results, ``bse``. d_fittedvalues_scaled same as d_fittedvalues but scaled by the standard errors of a predicted mean of the response. d_linpred local change in linear prediction. d_linpred_scale local change in linear prediction scaled by the standard errors for the prediction based on cov_params. Notes ----- This has not yet been tested for correctness when offset or exposure are used, although they should be supported by the code. Some GLM specific measures like d_deviance are still missing. Computing an explicit leave-one-observation-out (LOOO) loop is included but no influence measures are currently computed from it. cbt|dr|jS|jS)z Diagonal of the hat_matrix for GLM Notes ----- This returns the diagonal of the hat matrix that was provided as argument to GLMInfluence or computes it using the results method `get_hat_matrix`. r)rorr5get_hat_matrixrs r;rqzGLMInfluence.hat_matrix_diag@s4 4+ , , 1( (<..00 0r=ctj|j|jz}|tjd|jz z}|jS)zChange in parameter estimates Notes ----- This uses one-step approximation of the parameter change to deleting one observation. r)r&rrrArrDrqrrs r;rzGLMInfluence.d_paramsPsD **T-CC"'!d22333xr=c*tjS)a# Internally studentized pearson residuals Notes ----- residuals / sqrt( scale * (1 - hii)) where residuals are those provided to GLMInfluence which are pearson residuals by default, and hii is the diagonal of the hat matrix. )superr)rur#s r;rzGLMInfluence.resid_studentized_sww((r=c|j}|jdz|jz }||d|z z z}ddlm}|j||j|jj}||fS)aCook's distance Notes ----- Based on one step approximation using resid_studentized and hat_matrix_diag for the computation. Cook's distance divides by the number of explanatory variables. Computed using formulas for GLM and does not use results.cov_params. It includes p-values based on the F-distribution which are only approximate outside of linear Gaussian models. rrrrr-r.s r;rzGLMInfluence.cooks_distancepsn")Q.<C1s7O# 8T[$,2GHHr=c^|jjj}||jzdS)z Change in linear prediction This uses one-step approximation of the parameter change to deleting one observation ``d_params``. r)r5r rArr)rurAs r; d_linpredzGLMInfluence.d_linpreds,|!&t}$))!,,,r=c4|j|jjjz S)a  Change in linpred scaled by standard errors This uses one-step approximation of the parameter change to deleting one observation ``d_params``, and divides by the standard errors for linpred provided by results.get_prediction. )rrlinpredrrs r;d_linpred_scaledzGLMInfluence.d_linpred_scaleds~ 4 < ???r=ctjdtjjjt jfdtj D}| S)experimental code z this ignores offset and exposurecbg|]+\}}jj||,Sr)r5r predict)rrpirArus r;rz2GLMInfluence._fittedvalues_one..sLDDD$q" <-55b$q'BBDDDr=) rrrsrr5r rAr&array enumeratersqueeze)rufittedrAs` @r;_fittedvalues_onezGLMInfluence._fittedvalues_ones  8+FFF|!&DDDDD(1$/(B(BDDDEE~~r=cD|j|jz S)r)r5rrrs r;_diff_fittedvalues_onez#GLMInfluence._diff_fittedvalues_ones |##%%(>>>r=c (ddlm}d}|jjj}|jjj}|jj}|d}|d}|dx}} |dx} } |dd } t|d d rd } nd } tj |j t}tj |j t}tj |j t}||j }|D]\}}|||} | | |} | | ||d<|j||||f| | ||||d|}| r|d|j_||jjd}|j||<|j||<||||<t+||||S)acollect required results from the LOOO loop all results will be attached. currently only 'params', 'mse_resid', 'det_cov_params' are stored Reestimates the model with endog and exog dropping one observation at a time This uses a nobs loop, only attributes of the results instance are stored. Warning: This will need refactoring and API changes to be able to add options. rrPcdtj|Srr_r`s r;z(GLMInfluence._res_looo..ss~~7G7G)H)Hr= freq_weights var_weightsoffsetexposuren_trialsNfamily initializeTFrb)rrrrnewton) start_paramsr)rrrr?)rRrQr5r rHrA_get_init_kwdspopror&rdrBrerrrnrIrryrr)rurQrarHrA init_kwdsrrroffset_r exposure_r is_binomialrrr?rVrXrYmod_ir[s r;r<zGLMInfluence._res_looosL DCCCCCHH "(|!&L&5577  }}^44 mmM22 $==222(}}Z8889==T22 9X& 5 5 KKK$*E222E222%+U;;;+di(($ ? ?ME6! -#$UO #(0 *%$D$U5\4;2,3.72>u2E1%-//E"\..00F6N!KE&M%7%7%>%>N6 " "6%#1333 3r=)r$rrrr rqrrrr rrrrr< __classcell__)r#s@r;rrs1((T 1 1^ 1  ^  ) ) ) )^ ) ^6 - -X - @ @X @   X  ??X? B3B3^B3B3B3B3B3r=r)r r NFN)r>)r )rrrstatsmodels.compat.pandasrstatsmodels.compat.pythonr collectionsrnumpyr&)statsmodels.graphics._regressionplots_docr#statsmodels.regression.linear_modelrstatsmodels.stats.multitestrstatsmodels.tools.decoratorsr statsmodels.tools.toolsr r<rSr]r_rr rrrr=r;rs......******######IIIIII333333555555777777888888BF%)MMMMd)$)$)$)$X...bnnnnnnnnb~)~)~)~)~)&~)~)~)B F.F.F.F.F.&F.F.F.RNNNNbh3h3h3h3h3<h3h3h3h3h3r=