M/PhCdZddlmZddlZddlmZddlmZm Z m Z ddl m Z Gdde Z Gd d e Zdd Zdd Zd ZddZdS)alinear model with Theil prior probabilistic restrictions, generalized Ridge Created on Tue Dec 20 00:10:10 2011 Author: Josef Perktold License: BSD-3 open issues * selection of smoothing factor, strength of prior, cross validation * GLS, does this really work this way * None of inherited results have been checked yet, I'm not sure if any need to be adjusted or if only interpretation changes One question is which results are based on likelihood (residuals) and which are based on "posterior" as for example bse and cov_params * helper functions to construct priors? * increasing penalization for ordered regressors, e.g. polynomials * compare with random/mixed effects/coefficient, like estimated priors there is something fishy with the result instance, some things, e.g. normalized_cov_params, do not look like they update correctly as we search over lambda -> some stale state again ? I added df_model to result class using the hatmatrix, but df_model is defined in model instance not in result instance. -> not clear where refactoring should occur. df_resid does not get updated correctly. problem with definition of df_model, it has 1 subtracted for constant )lrangeN)cache_readonly)OLSGLSRegressionResults)atleast_2dcolsc8eZdZdZ d fd Zd dZd d ZxZS) TheilGLSa GLS with stochastic restrictions TheilGLS estimates the following linear model .. math:: y = X \beta + u using additional information given by a stochastic constraint .. math:: q = R \beta + v :math:`E(u) = 0`, :math:`cov(u) = \Sigma` :math:`cov(u, v) = \Sigma_p`, with full rank. u and v are assumed to be independent of each other. If :math:`E(v) = 0`, then the estimator is unbiased. Note: The explanatory variables are not rescaled, the parameter estimates not scale equivariant and fitted values are not scale invariant since scaling changes the relative penalization weights (for given \Sigma_p). Note: GLS is not tested yet, only Sigma is identity is tested Notes ----- The parameter estimates solves the moment equation: .. math:: (X' \Sigma X + \lambda R' \sigma^2 \Sigma_p^{-1} R) b = X' \Sigma y + \lambda R' \Sigma_p^{-1} q :math:`\lambda` is the penalization weight similar to Ridge regression. If lambda is zero, then the parameter estimate is the same as OLS. If lambda goes to infinity, then the restriction is imposed with equality. In the model `pen_weight` is used as name instead of $\lambda$ R does not have to be square. The number of rows of R can be smaller than the number of parameters. In this case not all linear combination of parameters are penalized. The stochastic constraint can be interpreted in several different ways: - The prior information represents parameter estimates from independent prior samples. - We can consider it just as linear restrictions that we do not want to impose without uncertainty. - With a full rank square restriction matrix R, the parameter estimate is the same as a Bayesian posterior mean for the case of an informative normal prior, normal likelihood and known error variance Sigma. If R is less than full rank, then it defines a partial prior. References ---------- Theil Goldberger Baum, Christopher slides for tgmixed in Stata (I do not remember what I used when I first wrote the code.) Parameters ---------- endog : array_like, 1-D dependent or endogenous variable exog : array_like, 1D or 2D array of explanatory or exogenous variables r_matrix : None or array_like, 2D array of linear restrictions for stochastic constraint. default is identity matrix that does not penalize constant, if constant is detected to be in `exog`. q_matrix : None or array_like mean of the linear restrictions. If None, the it is set to zeros. sigma_prior : None or array_like A fully specified sigma_prior is a square matrix with the same number of rows and columns as there are constraints (number of rows of r_matrix). If sigma_prior is None, a scalar or one-dimensional, then a diagonal matrix is created. sigma : None or array_like Sigma is the covariance matrix of the error term that is used in the same way as in GLS. Nct||||tj|}n] |jj}n#t $rd}YnwxYw|jd}tj|}|t|} | |=|| }|j\} }||_ ||j jdkrtd|t||_n#tj| dddf|_|jj| dfkrtd|vtj|}tj|dkr*tj|tj| z}n4|jdkrtj|}ntj| }|j| | fkrtd||_tj||_dS)Nsigmaz8r_matrix needs to have the same number of columnsas exogzq_matrix has wrong shapezsigma_prior has wrong shape)super__init__npasarraydata const_idxAttributeErrorshapeeyerr_matrixexog ValueErrorrq_matrixzerossizediagonesndim sigma_priorlinalgpinvsigma_prior_inv) selfendogrrrr!r rk_exogkeep_idx k_constraints __class__s h/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/sandbox/regression/penalized.pyrzTheilGLS.__init__|s E222  z(++HH ! I/ ! ! ! !  !Z]Fvf~~H$!&>>Y'#H- ( v  TY_Q' ' ''(( (  *844DMMH]33AAAtG > >:;; ;&!y~~k::s A AA?sandwichTc |}t|j|j|j}||_|j}|j}|j}|j } |j } |j dddf} tj | j| } | ||ztj |jtj | |zz} tj | j| ||ztj |jtj | |zz}tj| d}| |  |}tj ||}tj|}|dkr|}n|dkr|}nt%d||_||_||_t-||||}||_|S) aEstimate parameters and return results instance Parameters ---------- pen_weight : float penalization factor for the restriction, default is 1. cov_type : str, 'data-prior' or 'sandwich' 'data-prior' assumes that the stochastic restriction reflects a previous sample. The covariance matrix of the parameter estimate is in this case the same form as the one of GLS. The covariance matrix for cov_type='sandwich' treats the stochastic restriction (R and q) as fixed and has a sandwich form analogously to M-estimators. Returns ------- results : TheilRegressionResults instance Notes ----- cov_params for cov_type data-prior, is calculated as .. math:: \sigma^2 A^{-1} cov_params for cov_type sandwich, is calculated as .. math:: \sigma^2 A^{-1} (X'X) A^{-1} where :math:`A = X' \Sigma X + \lambda \sigma^2 R' \Simga_p^{-1} R` :math:`\sigma^2` is an estimate of the error variance. :math:`\sigma^2` inside A is replaced by the estimate from the initial GLS estimate. :math:`\sigma^2` in cov_params is obtained from the residuals of the final estimate. The sandwich form of the covariance estimator is not robust to misspecified heteroscedasticity or autocorrelation. r NgKH9)rcondr-z data-priorz-cov_type has to be 'sandwich' or 'data-prior')normalized_cov_paramsuse_t)rr&rr fitres_gls mse_residrrr$wexogwendogrdotTr"r#squeezerr0xpxisigma2_eTheilRegressionResultspenalization_factor)r% pen_weightcov_typer1lambdr3r;rrr$xyxxxpxxpyr: xpxi_sandwichparamsr0lfits r+r2z TheilGLS.fitsNdj$)4:>>>BBDD $==. J K$  VAC^^ BF?H4U4U!V!VVWfQS!nn BF?H4U4U!V!VVWy~~c~22 ((.. c""F## z ! !$1 ! !  % %$( ! !LMM M%2"   %dF-B%QQQ$)  aicccH|i}fd}ddlm}|j||fi|}|S)a\find penalization factor that minimizes gcv or an information criterion Parameters ---------- method : str the name of an attribute of the results class. Currently the following are available aic, aicc, bic, gc and gcv. start_params : float starting values for the minimization to find the penalization factor `lambd`. Not since there can be local minima, it is best to try different starting values. optim_args : None or dict optimization keyword arguments used with `scipy.optimize.fmin` Returns ------- min_pen_weight : float The penalization factor at which the target criterion is (locally) minimized. Notes ----- This uses `scipy.optimize.fmin` as optimizer. NcJt|S)N)getattrr2)r@methodr%s r+get_icz*TheilGLS.select_pen_weight..get_ics488E??F33 3rIr)optimize)scipyrPfmin)r%rN start_params optim_argsrOrPr@s`` r+select_pen_weightzTheilGLS.select_pen_weightsc2  J  4 4 4 4 4 4 #""""" flAAjAA rI)NNNN)r,r-T)rJr,N)__name__ __module__ __qualname____doc__rr2rU __classcell__r*s@r+r r +sNN`=A)-.;.;.;.;.;.;`PPPPd''''''''rIr ceZdZfdZedZdZedZedZedZ dZ dZ xZ S) r<ctj|i||dz |_|jjjd|jz dz |_dS)Nrr)rrhatmatrix_tracedf_modelmodelr&rdf_resid)r%argskwdsr*s r+rzTheilRegressionResults.__init__0s]$'$''',,..2  (.q1DMAAE rIc|jj}|jjtj||jjjjzdS)adiagonal of hat matrix diag(X' xpxi X) where xpxi = (X'X + sigma2_e * lambd * sigma_prior)^{-1} Notes ----- uses wexog, so this includes weights or sigma - check this case not clear whether I need to multiply by sigmahalf, i.e. (W^{-0.5} X) (X' W X)^{-1} (W^{-0.5} X)' or (W X) (X' W X)^{-1} (W X)' projection y_hat = H y or in terms of transformed variables (W^{-0.5} y) might be wrong for WLS and GLS case r)r`r0r5rr7r8sum)r%r:s r+hatmatrix_diagz%TheilRegressionResults.hatmatrix_diag7sE.z/  26$ 0@0B#C#C#EEJJ1MMMrIc4|jS)ztrace of hat matrix )rfrer%s r+r^z&TheilRegressionResults.hatmatrix_traceTs"&&(((rIcV|jd||jz z dzz SNr,)r4r^nobsrhs r+gcvzTheilRegressionResults.gcv`s,~d&:&:&<&> #$uyy'@'@4'HH&&&66 HLLTZ=S1SU[!\!\]]  Y " "4:#9 : :y"--&"$$rIc0|jdz|jjz S)aa measure for the fraction of the data in the estimation result The share of the prior information is `1 - share_data`. Returns ------- share : float between 0 and 1 share of data defined as the ration between effective degrees of freedom of the model and the number (TODO should be rank) of the explanatory variables. r)r_r`rankrhs r+ share_dataz!TheilRegressionResults.share_datas  !TZ_44rI) rVrWrXrrrfr^rmrprJrrrZr[s@r+r<r<.sFFFFFNN^N8)))MM^MPP^P^%%%$5555555rIr<ctj|d|z z }||Stj||f}||dd|||zf<|S)Nr,)rrr)n_coeffsn_varspositionreducedfulls r+coef_restriction_meandiffrsZfXH,G ~x6*++.5QQQ(** *+ rIctj| }d|dd|f<t|}||=tj||d}||Stj|dz |f}||dd|||zf<|S)Nrr)axis)rrrtaker)rrrbase_idxrkeeprs r+coef_restriction_diffbasersvhGGAAAxK (  D Xggt!,,,G ~x!V,--.5QQQ(** *+ rIc|d|dzz zS)Nrrk)ds r+next_oddrs AE ?rIrc |dkrddg}d}n5|dkr/ddlm}t|dz}|||}t j|t j|t|z f}ddlm} | |t j|t|z dzj } || St j|dz |f} | | dd|||zf<| S)Nrrkr)misc)ndiv)r") rQrrcentral_diff_weightsr concatenaterlenr"toeplitzr8) rdegreerrr diff_coeffsn_pointsrdffr"rrs r+coef_restriction_diffseqrs{{1g  !FQJ''//v/FF .+rx3{;K;K0K'L'LM N NCooc28Hs;7G7G,G!,K#L#LMMOG~x!V,--.5QQQ(** *+ rI)Nr)Nrr)rNrr)rYstatsmodels.compat.pythonrnumpyrstatsmodels.tools.decoratorsr#statsmodels.regression.linear_modelrrr#statsmodels.regression.feasible_glsrr r<rrrrrrIr+rs.!!D-,,,,,777777KKKKKKKKKK>>>>>>zzzzzszzzFf5f5f5f5f5.f5f5f5V"rI