M/Ph^GdZddlmZmZddlZddlZddlm cm Z ddlm Z m Z ddZdd Zd Zd ZGd dZGddeZdZdZdZGddZdS)aTools for working with groups This provides several functions to work with groups and a Group class that keeps track of the different representations and has methods to work more easily with groups. Author: Josef Perktold, Author: Nathaniel Smith, recipe for sparse_dummies on scipy user mailing list Created on Tue Nov 29 15:44:53 2011 : sparse_dummies Created on Wed Nov 30 14:28:24 2011 : combine_indices changes: add Group class Notes ~~~~~ This reverses the class I used before, where the class was for the data and the group was auxiliary. Here, it is only the group, no data is kept. sparse_dummies needs checking for corner cases, e.g. what if a category level has zero elements? This can happen with subset selection even if the original groups where defined as arange. Not all methods and options have been tried out yet after refactoring need more efficient loop if groups are sorted -> see GroupSorted.group_iter )lrangelzipN)Index MultiIndex.Fc, t|trtj|}ntj|}|j}|jdk}|r[|jd}|jj stj |d}| d|jfg|jdz}n|}tj |dd\ }} |r) | d| |r fd  D} | | | fS| | fS) zIuse np.unique to get integer group indices for product, intersection C)orderrT) return_indexreturn_inversec g|]A}dgtdzzt|zBS%srjoinlentuple.0iiprefixsepunis \/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/tools/grouputils.py z#combine_indices..HsV   4&SV"4555rB   ) isinstancernp column_stackasarraydtypendimshapeflags c_contiguousarrayviewuniquereshape) groupsrr return_labelsdtis2dncolsgroups_uni_idxuni_invlabelrs `` @rcombine_indicesr7%sR&%  $((F## B K1 D  Q|( 1XfC000F++FL12V\!_DEEIgD59;;;C' .hhrll""2u--%         e++$$r Tctjtjjdkr dddfnjdkr|rt d|rytjdjdzkrtjdtj fdtjdDStj }tj t|gtjddz}t|D]'\}}|kd||<(|S)zsimple bincount version, again group : ndarray, integer assumed to be consecutive integers no dtype checking because I want to raise in that case uses loop over columns of x for comparison, simple python loop r Nr znot implemented yetrcPg|]"}tjdd|f#S)N)weights)r"bincount)rcolgroupxs rrzgroup_sums..jsD    E1QQQV9555   r )r"r$squeezer& ValueErrormaxr'pd factorizer*ranger,zerosrlist enumeratesum)r>r= use_bincountuniquesresultrcats`` r group_sumsrMPsr 1 A Ju   % % ' 'Ev{{ aaagJ ! ./// 6%==1qwqz> ) )L''*Ex      ,,      )E""3w<<.4 +<+<<== )) 0 0GB5C<,,Q//F2JJ r cttj|drtj|j|S|j|zS)zesum by groups given group dummy variable group_dummy can be either ndarray or sparse matrix N) data_util_is_using_ndarray_typer"dotT)r> group_dummys rgroup_sums_dummyrTws: ' T::!vac;'''s[  r cddlm}tjt |dz}tjt |tj}||||f}|S)aHcreate a sparse indicator from a group array with integer labels Parameters ---------- groups : ndarray, int, 1d (nobs,) an array of group indicators for each observation. Group levels are assumed to be defined as consecutive integers, i.e. range(n_groups) where n_groups is the number of group levels. A group level with no observations for it will still produce a column of zeros. Returns ------- indi : ndarray, int8, 2d (nobs, n_groups) an indicator array with one row per observation, that has 1 in the column of the group level for that observation Examples -------- >>> g = np.array([0, 0, 2, 1, 1, 2, 0]) >>> indi = dummy_sparse(g) >>> indi <7x3 sparse matrix of type '' with 7 stored elements in Compressed Sparse Row format> >>> indi.todense() matrix([[1, 0, 0], [1, 0, 0], [0, 0, 1], [0, 1, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0]], dtype=int8) current behavior with missing groups >>> g = np.array([0, 0, 2, 0, 2, 0]) >>> indi = dummy_sparse(g) >>> indi.todense() matrix([[1, 0, 0], [1, 0, 0], [0, 0, 1], [1, 0, 0], [0, 0, 1], [1, 0, 0]], dtype=int8) r)sparser )r%)scipyrVr"arangeronesint8 csr_matrix)r.rVindptrdataindis r dummy_sparser_sk\ Ys6{{1} % %F 73v;;bg . . .D   dFF3 4 4D Kr cFeZdZd dZdZdZddefdZdZd d Z d d Z dS)Grouprc||_t|\}}}|||c|_|_|_t |j|_d|_|j|_|jr|jdz|_dSdS)Nr=) namer7 group_intr4rrn_groups separatorr)selfr=rdrr4r5s r__init__zGroup.__init__s~ / 6 6Wg25gw. dhDH  i ; ,++DKKK , ,r c4tj|jSN)r"r;rerhs rcountsz Group.countss{4>***r c|j|j|jjdkrfdD}nfdD}|S)Nr c g|]A}dgtdzzt|zBSrrrs rrz Group.labels..sV$$$SXXtfSQ[[&8999U2YYF$$$r c g|] }d|zz S)r)rrrs rrz Group.labels..s"666BVdRi'666r )rrrgr&)rhr6rrrs @@@rlabelsz Group.labelssyhn 8a<<$$$$$$"$$$EE7666#666E r NFc|j}|'tt|}||=||}|j}|s-|dddf|dddfk|St |jS)zl drop_idx is only available if sparse=False drop_idx is supposed to index into uni N)rrrr=astyper_re)rhdrop_idxrVr%ridxr=s rdummyz Group.dummys h  S""CH c(C  0!!!T'Nc$'l2::5AA A// /r cht||jr|j}|||fSrk)r! __class__r=)rhothers r interactionzGroup.interactions3 eT^ , , KE~~tUm,,,r Tc0t||j|SN)rI)rMre)rhr>rIs rrMzGroup.group_sumss!T^,GGGGr ctt|}t||z |j|}|||jz }||fSr})floatrrMre)rhr>rInobsmeans_g x_demeaneds r group_demeanzGroup.group_demeansPSVV}}QXt~*688800 7""r rT) __name__ __module__ __qualname__rirmrrintrwr{rMrrqr rraras,,,,$+++   "%s0000&--- HHHH######r rac,eZdZdfd ZdZdZxZS) GroupSortedrc2t|j|||tjtj|ddz}tdg|z|t|gz|_ dS)Nrdrr ) superryrir"nonzerodifftolistrrgroupidx)rhr=rdrvrys rrizGroupSorted.__init__s} dnd##,,U,>>>z"'%..))!,Q.6688aS3Ys5zzl(:;; r c#HK|jD]\}}t||VdSrk)rslice)rhlowupps r group_iterzGroupSorted.group_iter s<  " "HCS// ! ! ! ! " "r cjtj|jdddf|z }||k}||S)areturn the index array for lagged values Warning: if k is larger then the number of observations for an individual, then no values for that individual are returned. TODO: for the unbalanced case, I should get the same truncation for the array with lag=0. From the return of lag_idx we would not know which individual is missing. TODO: do I want the full equivalent of lagmat in tsa? maxlag or lag or lags. not tested yet Nr )r"r$r)rhlaglag_idxmask_oks r lag_indiceszGroupSorted.lag_indicess=*T]++AAAqD1C7'>wr r)rrrrirr __classcell__)rys@rrrs[<<<<<< """       r rc|d}t|tttjt jt jfrdSdS)z Checks if the first item of an array-like object is also array-like If so, we have a MultiIndex and returns True. Else returns False. rTF)r!rFrr"ndarrayrBSeries DataFrame)r>items r_is_hierarchicalr$s: Q4D$ubj")R\JKKtur c&tj|gd|iS)Nnames)r from_tuplesindexrs r_make_hierarchical_indexr1s  !E7 8% 8 88r ct|j}ttt|fdt|DS)NcFg|]}dzdz|S)z group{0:0})format)ripads rrz'_make_generic_names..8s0 D D D[_S ( ( + + D D Dr )rrstrrD)rn_namesrs @r_make_generic_namesr5sM%+G c#g,, C D D D DU7^^ D D DDr ceZdZddZedZedZedZedZddZ dd Z dd Z dd Z dd Z ddZddZddZdZddZddZdS)GroupingNcJt|ttfr9|/t|dr||dn||_||_nt|rt|||_nt|||_|Rt|j}t|jdr|j|dn ||j_t|j|_ t|jj|_ d|_ dS)a$ index : index-like Can be pandas MultiIndex or Index or array-like. If array-like and is a MultipleIndex (more than one grouping variable), groups are expected to be in each row. E.g., [('red', 1), ('red', 2), ('green', 1), ('green', 2)] names : list or str, optional The names to use for the groups. Should be a str if only one grouping variable is used. Notes ----- If index is already a pandas Index then there is no copy. N set_namesTinplacer)r!rrhasattrrrrrrrrrnlevelsslicesrhrrs rrizGrouping.__init__<s eeZ0 1 1 - 5+..(OOE4O8888"'EKDJJ&& 65eUCC "5u555 }+DJ774:{33-J(((====',DJ$ OO 4:+,,  r c\t|jdr |jjS|jjS)Nlevshape)rrrr'rls r index_shapezGrouping.index_shapebs, 4:z * * $:& &:# #r ct|jdr |jjStj|jjS)Nlevels)rrrrB Categoricalrls rrzGrouping.levelsis6 4:x ( ( 5:$ $>$*--4 4r ct|jdd}|Ft|jdr |jj}n$t j|jjd}|S)Ncodesrr)getattrrrrrrBrr)rhrs rrrzGrouping.labelspsZ GT22 =tz8,, ? )tz228> r c|jjSrkrrls r group_nameszGrouping.group_names{s zr c8||j}t||}dS)z, Resets the index in-place. N)rrrs rreindexzGrouping.reindexs$ =$Eu%%r rc8j}tj|}|t jtrfd|D_dSfd|D_dS)z Sets the slices attribute to be a list of indices of the sorted groups for the first index level. I.e., self.slices[0] is the index where each observation is in the first (sorted) group. cTg|]$}j|d%S)levelr)r get_loc_level)rr>rrhs rrz'Grouping.get_slices..sC,,,  :33AU3CCAF,,,r cDg|]}j|Srq)rget_loc)rr>rhs rrz'Grouping.get_slices..s)AAAQ4:--a00AAAr N) rget_level_valuesr,r"r*sortr!rr)rhrr.s`` r get_sliceszGrouping.get_slicess,,U33::<<&!! dj* - - B,,,,,$*,,,DKKKBAAA&AAADKKKr cNtj|j||_dS)zi Sets the attribute counts to equal the bincount of the (integer-valued) labels. N)r"r;rrrmrhrs rcount_categorieszGrouping.count_categoriess k$+e"455 r Tc|s|j}|rrtjtt ||}|}|j|jstd|rAt |t |krtddSdS)z Sanity checksrzData is not be sortedzDuplicate index entriesN) rrBrrrrequals Exceptionr,)rh is_sortedr,rtest test_sorteds r check_indexzGrouping.check_indexs JE  9<s5zz 2 2%@@@D))++K:$$[%677 9 7888  ;5zzS0000 9::: ; ;00r c||j}tj|dr~|jdkr,t j||d}|}n,t j||}|d}tj ||jfStj |dr4|}| |}|}||jfSd}t|) aApplies a (potentially hierarchical) sort operation on a numpy array or pandas series/dataframe based on the grouping index or a user-supplied index. Returns an object of the same type as the original data as well as the matching (sorted) Pandas index. Nr T)rcopyrFrz7data must be a Numpy array or a Pandas Series/DataFrame) rrOrPr&rBr sort_indexrr"r*_is_using_pandasrr@)rhr]routmsgs rrz Grouping.sorts =JE  +D$ 7 7 "yA~~iE===nn&&l4u555nnUn338C==#)+ +  'd 3 3 "C++e$$C..""C > !KCS// !r c |jd|jkrtd||j|fi|}d|jvrt j|St j|S)zeApply function to each column, by group Assumes that the dataframe already has a proper indexrz/dataframe does not have the same shape as indexrr )r'rrgroupbyapplyr"ravelr*)rh dataframefunctionrkwargsrs rtransform_dataframezGrouping.transform_dataframesy ?1  * *MNN N2ie,,28FFvFF  >>8C== 8C== r c |jd|jkrtdtj||j}|j||fd|i|S)z0Apply function to each column, by group r+array does not have the same shape as indexrr)r'rrrBrrr)rhr*rrrrs rtransform_arrayzGrouping.transform_arraysk ;q>TY & &IJJ JLdj999 't' 82252*022 2r c tj|}|jd|jkrt d||g}|jD]J}|jdkr ||ddf}n|jdkr||}||||fi|Ktj |}| d|jdS)zApply function to each group. Similar to transform_array but does not coerce array to a DataFrame and back and only works on a 1D or 2D numpy array. function is called function(group, group_idx, **kwargs). rrrr Nr r) r"r$r'rrrrr&appendr*r-)rhr*rrr processedssubsets rtransform_sliceszGrouping.transform_slicess  5!! ;q>TY & &IJJ J e$$$  < >> g = np.array([0, 0, 2, 1, 1, 2, 0]) >>> indi = dummy_sparse(g) >>> indi <7x3 sparse matrix of type '' with 7 stored elements in Compressed Sparse Row format> >>> indi.todense() matrix([[1, 0, 0], [1, 0, 0], [0, 0, 1], [0, 1, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0]], dtype=int8) current behavior with missing groups >>> g = np.array([0, 0, 2, 0, 2, 0]) >>> indi = dummy_sparse(g) >>> indi.todense() matrix([[1, 0, 0], [1, 0, 0], [0, 0, 1], [1, 0, 0], [0, 0, 1], [1, 0, 0]], dtype=int8) N)r_rrr)rhrr^s rr_zGrouping.dummy_sparses$^DK.// r rk)NN)r)TTN)rrrripropertyrrrrrrrrrrrrrrrr_rqr rrr;sp$$$$L$$X$ 55X5 X  X &&&&BBBB 6666 ; ; ; ;""""4 ! ! ! !2222::::*000000r r)rrFr)__doc__statsmodels.compat.pythonrrnumpyr"pandasrBstatsmodels.tools.datatoolsr]rOrrr7rMrTr_rarrrrrrqr rrs832222222*********$$$$$$$$'%'%'%'%V$$$$N!!!444nE#E#E#E#E#E#E#E#P     %   B   999EEE nnnnnnnnnnr