M/Ph_dZddlZddlmZddlZddlZddlm Z ddl m Z ddl m Z mZmZmZddlmcmZddlmcmZddlmZGdd eZGd d eZGd d ejZejeedS)zL Created on Sat Aug 22 20:24:42 2015 Author: Josef Perktold License: BSD-3 N)Appender)CategoricalDtype)stats)ModelLikelihoodModelGenericLikelihoodModelGenericLikelihoodModelResults)cache_readonlyc`eZdZdZejZdfd ZdZddZ e dfd Z e j je j _dZd Zd Zd Zd ZddZddZdZeejjdZdZdZedZeejj dfd ZxZS) OrderedModelacOrdinal Model based on logistic or normal distribution The parameterization corresponds to the proportional odds model in the logistic case. The model assumes that the endogenous variable is ordered but that the labels have no numeric interpretation besides the ordering. The model is based on a latent linear variable, where we observe only a discretization. y_latent = X beta + u The observed variable is defined by the interval y = {0 if y_latent <= cut_0 1 of cut_0 < y_latent <= cut_1 ... K if cut_K < y_latent The probability of observing y=k conditional on the explanatory variables X is given by prob(y = k | x) = Prob(cut_k < y_latent <= cut_k+1) = Prob(cut_k - x beta < u <= cut_k+1 - x beta = F(cut_k+1 - x beta) - F(cut_k - x beta) Where F is the cumulative distribution of u which is either the normal or the logistic distribution, but can be set to any other continuous distribution. We use standardized distributions to avoid identifiability problems. Parameters ---------- endog : array_like Endogenous or dependent ordered categorical variable with k levels. Labels or values of endog will internally transformed to consecutive integers, 0, 1, 2, ... pd.Series with ordered Categorical as dtype should be preferred as it gives the order relation between the levels. If endog is not a pandas Categorical, then categories are sorted in lexicographic order (by numpy.unique). exog : array_like Exogenous, explanatory variables. This should not include an intercept. pd.DataFrame are also accepted. see Notes about constant when using formulas offset : array_like Offset is added to the linear prediction with coefficient equal to 1. distr : string 'probit' or 'logit', or a distribution instance The default is currently 'probit' which uses the normal distribution and corresponds to an ordered Probit model. The distribution is assumed to have the main methods of scipy.stats distributions, mainly cdf, pdf and ppf. The inverse cdf, ppf, is only use to calculate starting values. Notes ----- Status: experimental, core results are verified, still subclasses `GenericLikelihoodModel` which will change in future versions. The parameterization of OrderedModel requires that there is no constant in the model, neither explicit nor implicit. The constant is equivalent to shifting all thresholds and is therefore not separately identified. Patsy's formula specification does not allow a design matrix without explicit or implicit constant if there are categorical variables (or maybe splines) among explanatory variables. As workaround, statsmodels removes an explicit intercept. Consequently, there are two valid cases to get a design matrix without intercept when using formulas: - specify a model without explicit and implicit intercept which is possible if there are only numerical variables in the model. - specify a model with an explicit intercept which statsmodels will remove. Models with an implicit intercept will be overparameterized, the parameter estimates will not be fully identified, cov_params will not be invertible and standard errors might contain nans. The computed results will be dominated by numerical imprecision coming mainly from convergence tolerance and numerical derivatives. The model will raise a ValueError if a remaining constant is detected. Nprobitc n|dkrtj|_n|dkrtj|_n||_|t j|}||_|||\}}}tj ||fi|d}|s|j j dkr_t j |j d\} } | |_ | }t j |rd} t| nC|j j dkr3t!|dstd |j jd}g}|jd krtd ||| |jdz |_|j|_|j|j|jzz |_t4|_dS) Nr logitT)return_inversezFNaN in dependent variable detected. Missing values need to be removed. design_infoz2-dim endog not supportedrz+There should not be a constant in the model)k_levels)rnormdistrlogisticnpasarrayoffset _check_inputssuper__init__endogndimuniqueisnanany ValueErrorhasattrshape k_constant_initialize_labelsrk_extrak_varsdf_modelnobsdf_residOrderedResults results_class) selfrexogrrkwdslabels is_pandasrr indexmsg __class__s d/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/miscmodels/ordinal_model.pyrzOrderedModel.__init__ts H  DJJ g  DJJDJ  Z''F #'#5#5eT#B#B vy----- z!## " $*T J J J " 8F##''))*@C$S//)*A%%t]33B$%@AAA :+A. ?Q  JKK K :::}q(    T[4<%?@ +ct|jtjs#|jjd}t j|d}d}t|tjrt|j tru|j j st jdt|j}|jj}|jj}|dkrt'd||_d}|||fS)aHandle endog that is pandas Categorical. Checks if self.distrib is legal and provides Pandas ordered Categorical support for endog. Parameters ---------- endog : array_like Endogenous, dependent variable, 1-D. exog : array_like Exogenous, explanatory variables. Currently not used. Returns ------- endog : array_like or pandas Series If the original endog is a pandas ordered Categorical Series, then the returned endog are the ``codes``, i.e. integer representation of ordere categorical variable labels : None or list If original endog is pandas ordered Categorical Series, then the categories are returned. Otherwise ``labels`` is None. is_pandas : bool This is True if original endog is a pandas ordered Categorical Series and False otherwise. z# is not a scipy.stats distribution.NFznthe endog has ordered == False, risk of capturing a wrong order for the categories. ordered == True preferred.z5missing values in categorical endog are not supportedT) isinstancerr rv_continuousnamewarningswarnpdSeriesdtypesrdtypeorderedWarningvalues categoriescatcodesminr#)r/rr0r5r2r3 endog_names r7rzOrderedModel._check_inputss:$*e&9:: :?GGG  M#    eRY ' ' !%,(899 !{*+M#K#*+++ #Z 0 99;;"$$$&5666'  fi''r8c||_|t||_n||_|j|jj\|_|_n|jjddc|_|_dt|dd|ddD}|jHt|j |jkrtd|j |dS||j _ dS)NrcXg|]'\}}t|dzt|z(S)/)str).0xys r7 z3OrderedModel._initialize_labels..sGEEE#q!q66C<#a&&0EEEr8r:rz)something wrong with exog_names, too long)r2lenrr0r%r+r)rzip exog_names RuntimeErrorextenddataxnames)r/r2rthreshold_namess r7r'zOrderedModel._initialize_labelss  KKDMM$DM 9 %)Y_ "DIt{{%)Z%5a%8! "DIt{EE'*6#2#;qrr 'C'CEEE 9 4?##dk11"#NOOO O " "? 3 3 3 3 3.DI   r8c|dd}||}tj|g|R|dgd|} | jjdkrt |jtr |jj sd} t| |j j } | | | jd| _|| j_| S)N~r Intercept)rY drop_colsrzBOnly ordered pandas Categorical are supported as endog in formulasr)splitstripr from_formularrr;rCrrDr#rFrGr'argmaxrYynames) clsformularYsubsetr_argskwargsrKoriginal_endogmodelr5r2r6s r7rbzOrderedModel.from_formulas]]3''*0022 j)$$ J:>JJ;-JJBHJJ ; q ~35EFF &&,4 &+ oo%#*5F  $ $V , , ,+,,Q//EK *EJ  r8c6|j|S)agCdf evaluated at x. Parameters ---------- x : array_like Points at which cdf is evaluated. In the model `x` is the latent variable plus threshold constants. Returns ------- Value of the cumulative distribution function of the underlying latent variable evaluated at x. )rcdfr/rQs r7rmzOrderedModel.cdfz~~a   r8c6|j|S)abPdf evaluated at x Parameters ---------- x : array_like Points at which cdf is evaluated. In the model `x` is the latent variable plus threshold constants. Returns ------- Value of the probability density function of the underlying latent variable evaluated at x. )rpdfrns r7rqzOrderedModel.pdf#ror8c~tj||||z dS)aInterval probability. Probability that value is in interval (low, upp], computed as prob = cdf(upp) - cdf(low) Parameters ---------- low : array_like lower bound for interval upp : array_like upper bound for interval Returns ------- float or ndarray Probability that value falls in interval (low, upp] r)rmaximumrm)r/lowupps r7probzOrderedModel.prob3s/(z$((3--$((3--7;;;r8c||jdz d}tj|ddtj|ddf}tjtj g|tjgf}|S)aWtransformation of the parameters in the optimization Parameters ---------- params : nd_array Contains (exog_coef, transformed_thresholds) where exog_coef are the coefficient for the explanatory variables in the linear term, transformed threshold or cutoff points. The first, lowest threshold is unchanged, all other thresholds are in terms of exponentiated increments. Returns ------- thresh : nd_array Thresh are the thresholds or cutoff constants for the intervals. rN)rr concatenateexpcumsuminf)r/params th_paramsthreshs r7transform_threshold_paramsz'OrderedModel.transform_threshold_paramsIs$T]Q./001 2A2!# !"" !6!6!8999? 26'FRVH =>> r8c tj|ddtjtj|ddf}|S)aGobtain transformed thresholds from original thresholds or cutoffs Parameters ---------- params : ndarray Threshold values, cutoff constants for choice intervals, which need to be monotonically increasing. Returns ------- thresh_params : ndarrray Transformed threshold parameter. The first, lowest threshold is unchanged, all other thresholds are in terms of exponentiated increments. Transformed parameters can be any real number without restrictions. Nrr:)rrxlogdiff)r/r| thresh_paramss r7"transform_reverse_threshold_paramsz/OrderedModel.transform_reverse_threshold_paramsasL$rr (*rwvcrc{/C/C(D(D(FGG r8rvcH||}||||}|dkr|S|dddf}|dd|z }|dd|z }|dkr|||} | S|dvr||} | St d) a Predicted probabilities for each level of the ordinal endog. Parameters ---------- params : ndarray Parameters for the Model, (exog_coef, transformed_thresholds). exog : array_like, optional Design / exogenous data. If exog is None, model exog is used. offset : array_like, optional Offset is added to the linear prediction with coefficient equal to 1. If offset is not provided and exog is None, uses the model's offset if present. If not, uses 0 as the default value. which : {"prob", "linpred", "cumprob"} Determines which statistic is predicted. - prob : predicted probabilities to be in each choice. 2-dim. - linear : 1-dim linear prediction of the latent variable ``x b + offset`` - cumprob : predicted cumulative probability to be in choice k or lower Returns ------- predicted values : ndarray If which is "prob", then 2-dim predicted probabilities with observations in rows and one column for each category or level of the categorical dependent variable. If which is "cumprob", then "prob" ar cumulatively added to get the cdf at k, i.e. probability of observing choice k or lower. If which is "linpred", then the conditional prediction of the latent variable is returned. In this case, the return is one-dimensional. )r0rlinpredNr:rrv)cumcumprobz`which` is not available)r_linpredrvrmr#) r/r|r0rwhichr~xbrtrurvrs r7predictzOrderedModel.predictwsL0088 ]]6V] < < I  I 4[SbSkBQRRj2o F??99S#&&DK ( ( (hhsmmGN788 8r8cB||j}||j}n|d}|tj|}|Otj|}tj|}||d|jdz }ntj|j}|||z }|S)aiLinear prediction of latent variable `x b + offset`. Parameters ---------- params : ndarray Parameters for the model, (exog_coef, transformed_thresholds) exog : array_like, optional Design / exogenous data. Is exog is None, model exog is used. offset : array_like, optional Offset is added to the linear prediction with coefficient equal to 1. If offset is not provided and exog is None, uses the model's offset if present. If not, uses 0 as the default value. Returns ------- linear : ndarray 1-dim linear prediction given by exog times linear params plus offset. This is the prediction for the underlying latent variable. If exog and offset are None, then the predicted values are zero. Nrr)r0rrrdotrzerosr+)r/r|r0r_exog_paramsrs r7rzOrderedModel._linpreds. <9D~~  Z''F  Jt$$Ej((Gii(=4=1+<)=(= >??GGhty))G   v Gr8c||}||j}||jdz}||}||z }||z }||fS)a}Integration bounds for the observation specific interval. This defines the lower and upper bounds for the intervals of the choices of all observations. The bounds for observation are given by a_{k_i-1} - linpred_i, a_k_i - linpred_i where - k_i is the choice in observation i. - a_{k_i-1} and a_k_i are thresholds (cutoffs) for choice k_i - linpred_i is the linear prediction for observation i Parameters ---------- params : ndarray Parameters for the model, (exog_coef, transformed_thresholds) Return ------ low : ndarray Lower bounds for choice intervals of each observation, 1-dim with length nobs upp : ndarray Upper bounds for choice intervals of each observation, 1-dim with length nobs. r)rrr)r/r|r~ thresh_i_low thresh_i_upprrtrus r7_boundszOrderedModel._boundssa<0088dj) dj1n- ]]6 " "RRCxr8cP||SN) loglikeobssum)r/r|s r7loglikezOrderedModel.loglikes"v&&**,,,r8c||\}}|||}tj|dzS)aY Log-likelihood of OrderdModel for all observations. Parameters ---------- params : array_like The parameters of the model. Returns ------- loglike_obs : array_like The log likelihood for each observation of the model evaluated at ``params``. g#B ;)rrvrr)r/r|rtrurvs r7rzOrderedModel.loglikeobss@<<''Syyc""vdUl###r8c`||\}}|||}||}||}||z dddf}||dddfz}tj|ddddf |jz|ddddff}|S)zscore, first derivative of loglike for each observations This currently only implements the derivative with respect to the exog parameters, but not with respect to threshold parameters. Nr)rrvrqr column_stackr0) r/r|rtrurvpdf_upppdf_low score_factorsos r7 score_obs_zOrderedModel.score_obs_s<<''Syyc""((3--((3-- ')111d73 QQQW % _|AAArrE22TY>*111abb51344 r8cftj|jt|jz }|jtj|dd}||}tj tj |j |f}|S)aStart parameters for the optimization corresponding to null model. The threshold are computed from the observed frequencies and transformed to the exponential increments parameterization. The parameters for explanatory variables are set to zero. rr) rbincountrrTrppfcliprzrrxrr))r/freq start_ppfstart_threshold start_paramss r7rzOrderedModel.start_params8s{4:&&TZ8JNN274;;==!Q#?#?@@ AA)LL~rx '<'L!!!! !!! <<<,0,49494949l****X%%%NX$,455--65-$$$&@  X Xo!)**KL*++*r8r ceZdZdZdZedZedZedZedZ edZ dS) r-zResults class for OrderedModel This class inherits from GenericLikelihoodModelResults and not all inherited methods might be appropriate in this case. ctj|jj}t j|jj|d}t j|d|d}t j || tddj d}|S)z$*"2-7GGGN4<<>>#8#8#;#;.8$HHH  I$OOC00$(#(***+,FF1II  r8cN|jj}|j|S)zS Value of the loglikelihood of model without explanatory variables )rkrr)r/ params_nulls r7llnullzOrderedResults.llnullts$ j- z!!+...r8c&d|j|jz z S)zC McFadden's pseudo-R-squared. `1 - (llf / llnull)` r)llfrr/s r7 prsquaredzOrderedResults.prsquared}s 48DK'''r8c&d|j|jz zS)zM Likelihood ratio chi-squared statistic; `-2*(llnull - llf)` )rrrs r7llrzOrderedResults.llrs 4;)**r8cjtjj|j|jjS)z The chi-squared probability of getting a log-likelihood ratio statistic greater than llr. llr has a chi-squared distribution with degrees of freedom `df_model`. )r distributionschi2sfrrkr)rs r7 llr_pvaluezOrderedResults.llr_pvalues'"'**48TZ5FGGGr8cddlm}|jj}|}||d}|t j|jd|f}|S)u"probability residual Probability-scale residual is ``P(Y < y) − P(Y > y)`` where `Y` is the observed choice and ``y`` is a random variable corresponding to the predicted distribution. References ---------- Shepherd BE, Li C, Liu Q (2016) Probability-scale residuals for continuous, discrete, and censored data. The Canadian Journal of Statistics. 44:463–476. Li C and Shepherd BE (2012) A new residual for ordinal outcomes. Biometrika. 99: 473–480 r)prob_larger_ordinal_choicer) statsmodels.stats.diagnostic_genrrkrrrrr%)r/rrfittedr resid_probs r7rzOrderedResults.resid_probsh$ POOOOO   & &v . .q 1ryQ00%78 r8N) rrrrrr rrrrrrr8r7r-r-[s $//^/((^( ++^+ HH^H^r8r-ceZdZdS)rN)rrrrr8r7rrsDr8r) rr>statsmodels.compat.pandasrnumpyrpandasr@pandas.api.typesrscipyrstatsmodels.base.modelrrrr statsmodels.base.wrapperbasewrapperwrap#statsmodels.regression.linear_model regression linear_modellmstatsmodels.tools.decoratorsr r r-RegressionResultsWrapperrpopulate_wrapperrr8r7rs......------ (''''''''000000000777777{{{{{){{{|QQQQQ2QQQh     B7   +^<<<<