M/PhNSdZddlmZddlmZddlZdMdZdZdZd Z Gd d Z e d krdZ d Z dddgfdgdfgfZ eegdgdZe r%eegdejdZegggdggggdgdgdddgdZe e eeeZd ejd<dejd<ededeeejed eejed!eejdd"d#d$gfd%d&d'd(gfd)gd*fgfd+d,gfgfZgd-d.gd/d.gd0d1gggd2gd3d4ggd5gd6d.d7gggd8gd9Zed:ed;ed<DZed=d>d?d@dAdBdC e e eeeZd ejdD<dejdE<edFedeeejed eejed!eejedGee ej!edHeej"eej"edIeej#edJeej$ ej%gdKZ&ee'e&ej%gdLZ(ee'e(dSdS)Na Formulas -------- This follows mostly Greene notation (in slides) partially ignoring factors tau or mu for now, ADDED (if all tau==1, then runmnl==clogit) leaf k probability : Prob(k|j) = exp(b_k * X_k / mu_j)/ sum_{i in L(j)} (exp(b_i * X_i / mu_j) branch j probabilities : Prob(j) = exp(b_j * X_j + mu*IV_j )/ sum_{i in NB(j)} (exp(b_i * X_i + mu_i*IV_i) inclusive value of branch j : IV_j = log( sum_{i in L(j)} (exp(b_i * X_i / mu_j) ) this is the log of the denominator of the leaf probabilities L(j) : leaves at branch j, where k is child of j NB(j) : set of j and it's siblings Design ------ * splitting calculation transmission between returns and changes to instance.probs - probability for each leaf is in instance.probs - inclusive values and contribution of exog on branch level need to be added separately. handed up the tree through returns * question: should params array be accessed directly through `self.recursionparams[self.parinddict[name]]` or should the dictionary return the values of the params, e.g. `self.params_node_dict[name]`. The second would be easier for fixing tau=1 for degenerate branches. The easiest might be to do the latter only for the taus and default to 1 if the key ('tau_'+branchname) is not found. I also need to exclude tau for degenerate branches from params, but then I cannot change them from the outside for testing and experimentation. (?) * SAS manual describes restrictions on tau (though their model is a bit different), e.g. equal tau across sibling branches, fixed tau. The also allow linear and non-linear (? not sure) restriction on params, the regression coefficients. Related to previous issue, callback without access to the underlying array, where params_node_dict returns the actual params value would provide more flexibility to impose different kinds of restrictions. bugs/problems ------------- * singleton branches return zero to `top`, not a value I'm not sure what they are supposed to return, given the split between returns and instance.probs DONE * Why does 'Air' (singleton branch) get probability exactly 0.5 ? DONE TODO ---- * add tau, normalization for nested logit, currently tau is 1 (clogit) taus also needs to become part of params MOSTLY DONE * add effect of branch level explanatory variables DONE * write a generic multinomial logit that takes arbitrary probabilities, this would be the same for MNL, clogit and runmnl, delegate calculation of probabilities * test on actual data, - tau=1 replicate clogit numbers, - transport example from Greene tests 1-level tree and degenerate sub-trees - test example for multi-level trees ??? * starting values: Greene mentiones that the starting values for the nested version come from the (non-nested) MNL version. SPSS uses constant equal (? check transformation) to sample frequencies and zeros for slope coefficient as starting values for (non-nested) MNL * associated test statistics - (I do not think I will fight with the gradient or hessian of the log-like.) - basic MLE statistics can be generic - tests specific to the model (?) * nice printouts since I'm currently collecting a lot of information in the tree recursion and everything has names The only parts that are really necessary to get a functional nested logit are adding the taus (DONE) and the MLE wrapper class. The rest are enhancements. I added fake tau, one fixed tau for all branches. (OBSOLETE) It's not clear where the tau for leaf should be added either at original assignment of self.probs, or as part of the one-step-down probability correction in the bottom branches. The second would be cleaner (would make treatment of leaves and branches more symmetric, but requires that initial assignment in the leaf only does initialization. e.g self.probs = 1. ??? DONE added taus still todo: - tau for degenerate branches are not identified, set to 1 for MLE - rename parinddict to paramsinddict Author: Josef Perktold License : BSD (3-clause) )lrange)pprintNcddlm}tj|tj|z }||tj||}|S)agenerate integer random variables given probabilties useful because it can be used as index into any array or sequence type Parameters ---------- w : 1d array_like sequence of weights, probabilities. The weights are normalized to add to one. size : int or tuple of ints shape of output array Returns ------- rvs : array of shape given by size random variables each distributed according to the same discrete distribution defined by (normalized) w. Examples -------- >>> np.random.seed(0) >>> randintw([0.4, 0.4, 0.2], size=(2,6)) array([[1, 1, 1, 1, 1, 1], [1, 2, 2, 0, 1, 1]]) >>> np.bincount(randintw([0.6, 0.4, 0.0], size=3000))/3000. array([ 0.59566667, 0.40433333]) r)random) numpy.randomrnpcumsumsum searchsortedprodreshape)wsizerprvss n/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/sandbox/regression/treewalkerclass.pyrandintwrosf>$##### ! RVAYYA .. .. / / 7 7 = =C Jct|tr1|\}}|g}|D]$}|t|%|SgS)z walk tree to get list of branches Parameters ---------- tree : list of tuples tree as defined for RU2NMNL Returns ------- branch : list list of all branch names ) isinstancetupleextend getbranches)treenamesubtreeasts rrrsZ$ g F & &B HH[__ % % % % Irc6t|tr|\}}|g}g}t|dkr|g}ng}|D]T}t|\}}} |||||| U|||fSg|ggfS)a walk tree to get list of branches and list of leaves Parameters ---------- tree : list of tuples tree as defined for RU2NMNL Returns ------- branch : list list of all branch names leaves : list list of all leaves names r)rrlengetnodesr) rrrabaladegrblds rr"r"s"$ gV  w<<1  6DDD  BrllGAq! IIaLLL IIaLLL KKNNNN2t| vr>rc&eZdZdZdZdZddZdS)RU2NMNLaNested Multinomial Logit with Random Utility 2 parameterization Parameters ---------- endog : ndarray not used in this part exog : dict_like dictionary access to data where keys correspond to branch and leaf names. The values are the data arrays for the exog in that node. tree : nested tuples and lists each branch, tree or subtree, is defined by a tuple (branch_name, [subtree1, subtree2, ..., subtreek]) Bottom branches have as subtrees the list of leaf names. paramsind : dictionary dictionary that maps branch and leaf names to the names of parameters, the coefficients for exogs) Methods ------- get_probs Attributes ---------- branches leaves paramsnames parinddict Notes ----- endog needs to be encoded so it is consistent with self.leaves, which defines the columns for the probability array. The ordering in leaves is determined by the ordering of the tree. In the dummy encoding of endog, the columns of endog need to have the same order as self.leaves. In the integer encoding, the integer for a choice has to correspond to the index in self.leaves. (This could be made more robust, by handling the endog encoding internally by leaf names, if endog is defined as categorical variable with associated category level names.) c>|_|_|_|_d_i_i_i_i_i_ i_ t|\_ _ _tj _t#d|Ddj Dz_tj_dt+jD_fdjD_dt3jtjz_t3jtj_djj d<dS)Nch|] }|D]}| Sr/).0jis r z#RU2NMNL.__init__..s<$3$3$3!01$3$3+,%&$3$3$3$3rcg|]}d|zS)ztau_%sr/)r0bnames r z$RU2NMNL.__init__..sIII%X-IIIrci|]\}}|| Sr/r/)r0idxrs r z$RU2NMNL.__init__..s*;;; T$;;;rc4i|]\}}|fd|DS)c*g|]}j|Sr/) paramsidx)r0r1selfs rr6z/RU2NMNL.__init__...s <<.sJBBB#&1Q<<<*>*@*@BBB "BIc$2B.C.C$D$DD!xD,<(=(=>>12dn_--...rc|_jtjfdjD}|S)a obtain the probability array given an array of parameters This is the function that can be called by loglike or other methods that need the probabilities as function of the params. Parameters ---------- params : 1d array, (nparams,) coefficients and tau that parameterize the model. The required length can be obtained by nparams. (and will depend on the number of degenerate leaves - not yet) Returns ------- probs : ndarray, (nobs, nchoices) probabilities for all choices for each observation. The order is available by attribute leaves. See note in docstring of class c*g|]}j|Sr/)rE)r0leafr=s rr6z%RU2NMNL.get_probs..?s IIIT 4 0IIIr)rW calc_probrr arrayrL)r=params probs_arrays` r get_probszRU2NMNL.get_probs&sM. & ty!!!hIIIIT[IIIJJ rNc  |j}|j}|j}|j}t |t r|\}}g|j|<|j|jd|z} tr7tdt|||d| td|g} tdkrd}ntdkr ||}n|}|D]_} trt| | | |} tj| | z } | | || z}`| |j|<tr/tdt|td |||rBtrtd ||j||j| g|j|<t1|D]l\}} trtdt3| |t | t s[|j||j| |j| |z |j| <trtd|| d}t5|j|}tj||z dksJ| ||z }|j|||j|D]G}trtd|||||j|tj|dz|j|<Hntrtd||tdkr|S||j|<tj|j|dkr9tj|j||j|j|z}nd}|dks|j|jd|z} nd} || |zz}|| tj|zzS|j|jd|z} trtd ||j|||d zd d|j|zzg|j|<tjtj|j||j|j|z| z }||j|<tdkrtj|StdkrHtj||}tr%td|d|||Stdkrd|SdS)z5walking a tree bottom-up based on dictionary tau_z+----------- starting next branch-----------ztau=rr)rrz*----------- returning to branch-----------zbranchsum in branchparent*z-probz(%s)z, z"self.datadict[name], self.probs[k]zrepr(b)z&*********** branchsum at bottom branchg:0yE> branchprobg-C6?zworking on branchtopzfinal branch withr-N) rArBrCrDrrrGrWr<DEBUGprinttestxbr^r expappendrHrrElogrFjoinrrJrSreprr absmaximumrIrUr_)r=rrerArBrCrDrrtau branchvaluer&bvtmpsumr>ivbidxr5 branchsum2bprobbranchxbleafprob leavessums rr^zRU2NMNL.calc_probFs  =N N  dE " "G % MD'&(D d #&t~fTk'BCC *CDDDdHTNFC888i)))K{{ 1$TN   + +!HHH^^At,,VBsF^^""2&&&%N &1D d # >BCCCd +T9=== J,(F+++!&)001B41HIII F8!#DK $W-- P Pa4)T!WWd333!!e,,PK%,,TZ];;;$(JqMI$=DJqMSF RRRaDE!$T%6t% &4>$3G)H)H H$I#JDM$ vbfT]4%8"&"6tt7L"M&NOO #$%%H (DJt {{vh'''1HXd^44 O-tRWWT]]INNN  1wwt}}$r)N)__name__ __module__ __qualname____doc__rZrbr^r/rrr+r+sY))V&3&3&3T@S%S%S%S%S%S%rr+__main__rhFlyAirGround)TrainCarBus)rrrr)Airdata TraindataBusdataCardata)rhrr)GCTtmeConstAHinc)rrConstT)rrConstBrr)rhrrrrrrz Example 1z --------- Treez modru.probsB1rr&B2B21cr(B22)efgB3hconstarconstbconst2x2)constcrtime)constdrrx22)consterhince)constfrhincfhincg)consthrr)rrr&rrrr(rrrrrrrhcg|]}|Sr/r/)r0r2s rr6r64s;;;Aa;;;rabcdefghidi,)rhrrrrrz Example 2z sum of probsrHzbranch probabilitieszdegenerate branches) r@rrrrr@r@@)rrrr@rrrrrrrrrrrrr@r@r@rr@r@)r))rstatsmodels.compat.pythonrrnumpyr rrr"rkr+r~rirAtree0dictziprBrVupdaterCmodrurWrjr^rrEtree2 paramsind2 datadict2modru2r listrPrHrJrMr_params1rbparams2r/rrrsffN-,,,,,""""H.!!!H I%I%I%I%I%I%I%I%Z z E E3334 E  tCC666FFFHHIIH'4:::! ! &&'' OOr!!##$$$;;;333333 &> I GE8UI 6 6E !E" E! E+ E- E%//%* % %&&& E&MMM F5: E/ F5; Cy!Sz*/ u  E  S/ S/ d   ! ! ! ! ! ! G " " " " " "' "    J";;Zq !:!:;;;<