
    M/Ph                       d Z ddlZddlZddlZddlZddlmZ ddl	m
Z
 ddlmZ ddlmc mZ ddlmZ ddlmZ ddlmZ d	Zd
 Zd Zd Z G d d          Zd Z G d d          Zd Zd Zd Z  G d dej!                  Z" G d d          Z# G d dej$        ej%                  Z& G d dej'                  Z(d Z)dS )a  
Linear mixed effects models are regression models for dependent data.
They can be used to estimate regression relationships involving both
means and variances.

These models are also known as multilevel linear models, and
hierarchical linear models.

The MixedLM class fits linear mixed effects models to data, and
provides support for some common post-estimation tasks.  This is a
group-based implementation that is most efficient for models in which
the data can be partitioned into independent groups.  Some models with
crossed effects can be handled by specifying a model with a single
group.

The data are partitioned into disjoint groups.  The probability model
for group i is:

Y = X*beta + Z*gamma + epsilon

where

* n_i is the number of observations in group i

* Y is a n_i dimensional response vector (called endog in MixedLM)

* X is a n_i x k_fe dimensional design matrix for the fixed effects
  (called exog in MixedLM)

* beta is a k_fe-dimensional vector of fixed effects parameters
  (called fe_params in MixedLM)

* Z is a design matrix for the random effects with n_i rows (called
  exog_re in MixedLM).  The number of columns in Z can vary by group
  as discussed below.

* gamma is a random vector with mean 0.  The covariance matrix for the
  first `k_re` elements of `gamma` (called cov_re in MixedLM) is
  common to all groups.  The remaining elements of `gamma` are
  variance components as discussed in more detail below. Each group
  receives its own independent realization of gamma.

* epsilon is a n_i dimensional vector of iid normal
  errors with mean 0 and variance sigma^2; the epsilon
  values are independent both within and between groups

Y, X and Z must be entirely observed.  beta, Psi, and sigma^2 are
estimated using ML or REML estimation, and gamma and epsilon are
random so define the probability model.

The marginal mean structure is E[Y | X, Z] = X*beta.  If only the mean
structure is of interest, GEE is an alternative to using linear mixed
models.

Two types of random effects are supported.  Standard random effects
are correlated with each other in arbitrary ways.  Every group has the
same number (`k_re`) of standard random effects, with the same joint
distribution (but with independent realizations across the groups).

Variance components are uncorrelated with each other, and with the
standard random effects.  Each variance component has mean zero, and
all realizations of a given variance component have the same variance
parameter.  The number of realized variance components per variance
parameter can differ across the groups.

The primary reference for the implementation details is:

MJ Lindstrom, DM Bates (1988).  "Newton Raphson and EM algorithms for
linear mixed effects models for repeated measures data".  Journal of
the American Statistical Association. Volume 83, Issue 404, pages
1014-1022.

See also this more recent document:

http://econ.ucsb.edu/~doug/245a/Papers/Mixed%20Effects%20Implement.pdf

All the likelihood, gradient, and Hessian calculations closely follow
Lindstrom and Bates 1988, adapted to support variance components.

The following two documents are written more from the perspective of
users:

http://lme4.r-forge.r-project.org/lMMwR/lrgprt.pdf

http://lme4.r-forge.r-project.org/slides/2009-07-07-Rennes/3Longitudinal-4.pdf

Notation:

* `cov_re` is the random effects covariance matrix (referred to above
  as Psi) and `scale` is the (scalar) error variance.  For a single
  group, the marginal covariance matrix of endog given exog is scale*I
  + Z * cov_re * Z', where Z is the design matrix for the random
  effects in one group.

* `vcomp` is a vector of variance parameters.  The length of `vcomp`
  is determined by the number of keys in either the `exog_vc` argument
  to ``MixedLM``, or the `vc_formula` argument when using formulas to
  fit a model.

Notes:

1. Three different parameterizations are used in different places.
The regression slopes (usually called `fe_params`) are identical in
all three parameterizations, but the variance parameters differ.  The
parameterizations are:

* The "user parameterization" in which cov(endog) = scale*I + Z *
  cov_re * Z', as described above.  This is the main parameterization
  visible to the user.

* The "profile parameterization" in which cov(endog) = I +
  Z * cov_re1 * Z'.  This is the parameterization of the profile
  likelihood that is maximized to produce parameter estimates.
  (see Lindstrom and Bates for details).  The "user" cov_re is
  equal to the "profile" cov_re1 times the scale.

* The "square root parameterization" in which we work with the Cholesky
  factor of cov_re1 instead of cov_re directly.  This is hidden from the
  user.

All three parameterizations can be packed into a vector by
(optionally) concatenating `fe_params` together with the lower
triangle or Cholesky square root of the dependence structure, followed
by the variance parameters for the variance components.  The are
stored as square roots if (and only if) the random effects covariance
matrix is stored as its Cholesky factor.  Note that when unpacking, it
is important to either square or reflect the dependence structure
depending on which parameterization is being used.

Two score methods are implemented.  One takes the score with respect
to the elements of the random effects covariance matrix (used for
inference once the MLE is reached), and the other takes the score with
respect to the parameters of the Cholesky square root of the random
effects covariance matrix (used for optimization).

The numerical optimization uses GLS to avoid explicitly optimizing
over the fixed effects parameters.  The likelihood that is optimized
is profiled over both the scale parameter (a scalar) and the fixed
effects parameters (if any).  As a result of this profiling, it is
difficult and unnecessary to calculate the Hessian of the profiled log
likelihood function, so that calculation is not implemented here.
Therefore, optimization methods requiring the Hessian matrix such as
the Newton-Raphson algorithm cannot be used for model fitting.
    N)sparse)norm)Penalty)data)cache_readonly)ConvergenceWarningz1The random effects covariance matrix is singular.c                 Z   t          | t          j                  r/t          |t          j                  rt          j        | |          S t	          j        |           r|                     |          S t	          j        |          r$|j                            | j                  j        S dS )zL
    Returns the dot product of the arrays, works for sparse and dense.
    N)
isinstancenpndarraydotr   issparseTxys     i/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/regression/mixed_linear_model.py_dotr      s    
 !RZ   Z2:%>%> va||			 uuQxx			 swwqs||~     c                    | j         d         | j         d         z  |j         d         z  | j         d         |j         d         z  |j         d         z  z   }|j         d         |j         d         z  |j         d         z  | j         d         | j         d         z  |j         d         z  z   }||k     rt          t          | |          |          S t          | t          ||                    S )z
    Find best ordering for three arrays and do the multiplication.

    Doing in manually instead of using dynamic programing is
    approximately 15 times faster.
    r      )shaper   )ABCcost1cost2s        r   _multi_dot_threer      s     WQZ!'!*$qwqz1WQZ!'!*$qwqz12E WQZ!'!*$qwqz1WQZ!'!*$qwqz12E u}}DAJJ"""AtAqzz"""r   c                     t          j        |           r'|                     |                                          S t	          j        |                                 |                                          S )zy
    Returns sum(x * y), where '*' is the pointwise product, computed
    efficiently for dense and sparse matrices.
    )r   r   multiplysumr   r   ravelr   s     r   _dotsumr#      sV     q ,zz!}}  """ vaggii+++r   c                       e Zd ZdZd ZdS )VCSpeca  
    Define the variance component structure of a multilevel model.

    An instance of the class contains three attributes:

    - names : names[k] is the name of variance component k.

    - mats : mats[k][i] is the design matrix for group index
      i in variance component k.

    - colnames : colnames[k][i] is the list of column names for
      mats[k][i].

    The groups in colnames and mats must be in sorted order.
    c                 0    || _         || _        || _        d S N)namescolnamesmats)selfr(   r)   r*   s       r   __init__zVCSpec.__init__   s    
 			r   N)__name__
__module____qualname____doc__r,    r   r   r%   r%      s-              r   r%   c                 N   | j         dk    rg S t          |t          j                  r|j                                        S t          |t          j                  r|j        |j        gS t          |t                    r|S d t          |j
        d                   D             }|S )z
    Passes through if given a list of names. Otherwise, gets pandas names
    or creates some generic variable names as needed.
    r   Nc                      g | ]}d |dz   dS )x_rer   1dr1   .0ks     r   
<listcomp>z&_get_exog_re_names.<locals>.<listcomp>   s(    CCCa!q1u!!!CCCr   r   )k_rer
   pd	DataFramecolumnstolistSeriesnamelistranger   )r+   exog_redefnamess      r   _get_exog_re_namesrE      s    
 yA~~	'2<(( %%'''	GRY	'	' GL,D~	GT	"	"  DC5q1A+B+BCCCHOr   c                   d    e Zd ZdZd Zd Z ee          Z	 	 d	dZ ee          Zd Zd
dZ	dS )MixedLMParamsa  
    This class represents a parameter state for a mixed linear model.

    Parameters
    ----------
    k_fe : int
        The number of covariates with fixed effects.
    k_re : int
        The number of covariates with random coefficients (excluding
        variance components).
    k_vc : int
        The number of variance components parameters.

    Notes
    -----
    This object represents the parameter state for the model in which
    the scale parameter has been profiled out.
    c                     || _         || _        ||dz   z  dz  | _        || _        | j         | j        z   | j        z   | _        t          j        | j                  | _        d S )Nr      )k_fer:   k_re2k_vck_totr   tril_indices_ix)r+   rJ   r:   rL   s       r   r,   zMixedLMParams.__init__  sZ    		TAX&!+
	Y+di7
?49--r   c                    t          ||dz   z  dz            }|rt          |           |z
  |z
  }nt          |           |z
  }t          |||          }t          j        ||f          }|j        }	|r | d|         |_        | |||z            ||	<   n&t          j        |          |_        | d|         ||	<   |rt          j        ||j                  }n1||j        z   t          j	        t          j	        |                    z
  }||_
        |dk    r'|r| | d         dz  |_        n*| | d         |_        nt          j        g           |_        |S )a=  
        Create a MixedLMParams object from packed parameter vector.

        Parameters
        ----------
        params : array_like
            The mode parameters packed into a single vector.
        k_fe : int
            The number of covariates with fixed effects
        k_re : int
            The number of covariates with random effects (excluding
            variance components).
        use_sqrt : bool
            If True, the random effects covariance matrix is provided
            as its Cholesky factor, otherwise the lower triangle of
            the covariance matrix is stored.
        has_fe : bool
            If True, `params` contains fixed effects parameters.
            Otherwise, the fixed effects parameters are set to zero.

        Returns
        -------
        A MixedLMParams object.
        r   rI   r   N)intlenrG   r   zerosrO   	fe_paramsr   r   diagcov_revcomparray)
paramsrJ   r:   use_sqrthas_ferK   rL   parV   ixs
             r   from_packedzMixedLMParams.from_packed  sb   2 DD1H%)**  	'v;;%-DDv;;&D4t,,4,''V 	)!!D&>BLT%Z0F2JJ8D>>BL%F2J 	DVFFH--FFvx'27276??+C+CCF	!88 *!4%&&>1,!4%&&>x||BH	r   Nc                    |t          j        d          }| t          j        d          } ||t          j        d          }t          |           }t          |          }||j        d         n|j        d         }t	          |||          }| |_        | t          j        ||j                  |_        n	|||_        ||_	        |S )a  
        Create a MixedLMParams object from each parameter component.

        Parameters
        ----------
        fe_params : array_like
            The fixed effects parameter (a 1-dimensional array).  If
            None, there are no fixed effects.
        cov_re : array_like
            The random effects covariance matrix (a square, symmetric
            2-dimensional array).
        cov_re_sqrt : array_like
            The Cholesky (lower triangular) square root of the random
            effects covariance matrix.
        vcomp : array_like
            The variance component parameters.  If None, there are no
            variance components.

        Returns
        -------
        A MixedLMParams object.
        Nr   r   r   )
r   emptyrR   r   rG   rT   r   r   rV   rW   )rT   rV   cov_re_sqrtrW   rJ   rL   r:   r\   s           r   from_componentszMixedLMParams.from_componentsV  s    2 =HQKKEI>k1Xf%%F9~~5zz"("4v|A+:KA:N4t,, "{KM::BIIBI	r   c                     t          | j        | j        | j                  }| j                                        |_        | j                                        |_        | j                                        |_        |S )z/
        Returns a copy of the object.
        )rG   rJ   r:   rL   rT   copyrV   rW   )r+   objs     r   re   zMixedLMParams.copy  s]     DIty$)<<++--[%%''
JOO%%	
r   Fc                 4   | j         dk    r|r	 t          j                            | j                  }nW# t          j        j        $ r@ t          j        t          j        t          j        | j                                      }Y nw xY w|| j                 }n'| j        | j                 }nt          j	        d          }|rt          j        | j
                  }n| j
        }|rt          j        | j        ||f          }nt          j        ||f          }|S )a  
        Return the model parameters packed into a single vector.

        Parameters
        ----------
        use_sqrt : bool
            If True, the Cholesky square root of `cov_re` is
            included in the packed result.  Otherwise the
            lower triangle of `cov_re` is included.
        has_fe : bool
            If True, the fixed effects parameters are included
            in the packed result, otherwise they are omitted.
        r   )r:   r   linalgcholeskyrV   LinAlgErrorrU   sqrtrO   rS   rW   concatenaterT   )r+   rZ   r[   LcparW   r\   s          r   
get_packedzMixedLMParams.get_packed  s    9q== ,?	**4;77AAy, ? ? ?(<(< = =>>AAA?kk$(+(1++C 	GDJ''EEJE 	.e <==BBe--B	s   $4 ABB)NNNN)F)
r-   r.   r/   r0   r,   r^   staticmethodrc   re   ro   r1   r   r   rG   rG      s         &. . .: : :x ,{++KAE"- - - -^ #l?33O  % % % % % %r   rG   c                    	 | z  |j         d         }d|d|fxx         |z  cc<   t          j                  r0|d|dfxx         t          j        |          z  cc<    fd}nZj         d         }j        ||dz   z  d|dz   xx         |z  cc<   t
          j                            j                  		 fd}|S )a  
    Returns a solver for the linear system:

    .. math::

        (sI + ABA^\prime) y = x

    The returned function f satisfies f(x) = y as defined above.

    B and its inverse matrix are block diagonal.  The upper left block
    of :math:`B^{-1}` is Qi and its lower right block is diag(di).

    Parameters
    ----------
    s : scalar
        See above for usage
    A : ndarray
        p x q matrix, in general q << p, may be sparse.
    AtA : square ndarray
        :math:`A^\prime  A`, a q x q matrix.
    Qi : square symmetric ndarray
        The matrix `B` is q x q, where q = r + d.  `B` consists of a r
        x r diagonal block whose inverse is `Qi`, and a d x d diagonal
        block, whose inverse is diag(di).
    di : 1d array_like
        See documentation for Qi.

    Returns
    -------
    A function for solving a linear system, as documented above.

    Notes
    -----
    Uses Sherman-Morrison-Woodbury identity:
        https://en.wikipedia.org/wiki/Woodbury_matrix_identity
    r   Nc                     j                             |           }t          j                            |          }|j        | j        k     r|d d d f         }                    |          }| z  |dz  z  z
  S NrI   )r   r   r   rh   spsolvendim)rhsqlr   qmatss     r   solverz_smw_solver.<locals>.solver  ss    B &&tR00Bw!!4[rB7R!Q$Y&&r   r   c                 t    t          j        |           }t          j        |          }| z  |dz  z  z
  S rs   )r   r   )rv   rw   r   qmatiry   s     r   rz   z_smw_solver.<locals>.solver  s<     s##B2B7R!Q$Y&&r   )	r   r   r   diagsflatr   rh   solver   )
ry   r   AtAQidimrz   drx   r|   s
   ``      @@r   _smw_solverr     s#   N 7D
A1acNNNbNNNq 'QRRVR(((		' 		' 		' 		' 		' 		' 		' 		' JqM	!QqS',1Q3,2%	ac**	' 	' 	' 	' 	' 	' 	' Mr   c                    |j         d         }|t          j        |           z  }|| z  }|j         d         }	|d|	d|	fxx         |z  cc<   t          j        |          r||	d|	dfxx         t          j        |          z  cc<   t          j                            |          }
|
j        	                                
                    t          j                  }|
j        	                                
                    t          j                  }t          j        |                                          t          j        |                                          z   }|j        }nP|j         d         }|j        |	|dz   z  d|dz   xx         |z  cc<   t          j                            |          \  }}||z   |z   S )a  
    Returns the log determinant of

    .. math::

        sI + ABA^\prime

    Uses the matrix determinant lemma to accelerate the calculation.
    B is assumed to be positive definite, and s > 0, therefore the
    determinant is positive.

    Parameters
    ----------
    s : positive scalar
        See above for usage
    A : ndarray
        p x q matrix, in general q << p.
    AtA : square ndarray
        :math:`A^\prime  A`, a q x q matrix.
    Qi : square symmetric ndarray
        The matrix `B` is q x q, where q = r + d.  `B` consists of a r
        x r diagonal block whose inverse is `Qi`, and a d x d diagonal
        block, whose inverse is diag(di).
    di : 1d array_like
        See documentation for Qi.
    B_logdet : real
        The log determinant of B

    Returns
    -------
    The log determinant of s*I + A*B*A'.

    Notes
    -----
    Uses the matrix determinant lemma:
        https://en.wikipedia.org/wiki/Matrix_determinant_lemma
    r   Nr   )r   r   logr   r   r}   rh   splurm   diagonalastype
complex128Ur!   realr~   slogdet)ry   r   r   r   r   B_logdetpldrx   r   ludlduld1r   _s                   r   _smw_logdetr     s   N 	

A	
RVAYYB7D
A1acNNNbNNNt )QRRVR((( ]%%T]]__##BM22T]]__##BM22fRjjnn!1!11hJqM	!QqS',1Q3,2%""4((3b=3r   c           
      h  	
 g 
g g 	t                      }|                                 D ])\  }}|t          |                                          z  }*t          |          }|                                 |                                 D ]\  }}
                    |           g g }}|D ]}	 |                    ||         j                   nM# t          $ r@ |                    d t          ||         j	        d                   D                        Y nw xY w|                    ||                                        |           	                    |           t          j        
          }
fd|D             
fd|D             	fd|D             	t          
	          S )Nc                 ,    g | ]}t          |          S r1   )str)r7   js     r   r9   z_convert_vc.<locals>.<listcomp>T  s     F F FAQ F F Fr   r   c                      g | ]
}|         S r1   r1   )r7   ivc_namess     r   r9   z_convert_vc.<locals>.<listcomp>Z  s    ((((((r   c                      g | ]
}|         S r1   r1   )r7   r   vc_colnamess     r   r9   z_convert_vc.<locals>.<listcomp>[  s    ...a;q>...r   c                      g | ]
}|         S r1   r1   )r7   r   vc_matss     r   r9   z_convert_vc.<locals>.<listcomp>\  s    &&&awqz&&&r   )setitemskeysrA   sortappendr=   AttributeErrorrB   r   r   argsortr%   )exog_vcgroupsr8   vr)   r*   giir   r   r   s           @@@r   _convert_vcr   @  s   HKG UUF    1#affhh--&\\F
KKMMM 
 
1R$ 	 	AH!----! H H H F Fqtz!}1E1E F F FGGGGGHKK!8$$$t	H		B((((R(((H....2...K&&&&2&&&G(K111s   0 CADDc                        e Zd ZdZ	 	 d fd	Zd Ze	 	 d  fd	            Zd!d	Zd
 Z		 	 d"dZ
d#dZd Zd Zd Zd$dZd!dZd$dZd Zd$dZd Zd Z	 	 	 d% fd	Zd Z xZS )&MixedLMa	  
    Linear Mixed Effects Model

    Parameters
    ----------
    endog : 1d array_like
        The dependent variable
    exog : 2d array_like
        A matrix of covariates used to determine the
        mean structure (the "fixed effects" covariates).
    groups : 1d array_like
        A vector of labels determining the groups -- data from
        different groups are independent
    exog_re : 2d array_like
        A matrix of covariates used to determine the variance and
        covariance structure (the "random effects" covariates).  If
        None, defaults to a random intercept for each group.
    exog_vc : VCSpec instance or dict-like (deprecated)
        A VCSPec instance defines the structure of the variance
        components in the model.  Alternatively, see notes below
        for a dictionary-based format.  The dictionary format is
        deprecated and may be removed at some point in the future.
    use_sqrt : bool
        If True, optimization is carried out using the lower
        triangle of the square root of the random effects
        covariance matrix, otherwise it is carried out using the
        lower triangle of the random effects covariance matrix.
    missing : str
        The approach to missing data handling

    Notes
    -----
    If `exog_vc` is not a `VCSpec` instance, then it must be a
    dictionary of dictionaries.  Specifically, `exog_vc[a][g]` is a
    matrix whose columns are linearly combined using independent
    random coefficients.  This random term then contributes to the
    variance structure of the data for group `g`.  The random
    coefficients all have mean zero, and have the same variance.  The
    matrix must be `m x k`, where `m` is the number of observations in
    group `g`.  The number of columns may differ among the top-level
    groups.

    The covariates in `exog`, `exog_re` and `exog_vc` may (but need
    not) partially or wholly overlap.

    `use_sqrt` should almost always be set to True.  The main use case
    for use_sqrt=False is when complicated patterns of fixed values in
    the covariance structure are set (using the `free` argument to
    `fit`) that cannot be expressed in terms of the Cholesky factor L.

    Examples
    --------
    A basic mixed model with fixed effects for the columns of
    ``exog`` and a random intercept for each distinct value of
    ``group``:

    >>> model = sm.MixedLM(endog, exog, groups)
    >>> result = model.fit()

    A mixed model with fixed effects for the columns of ``exog`` and
    correlated random coefficients for the columns of ``exog_re``:

    >>> model = sm.MixedLM(endog, exog, groups, exog_re=exog_re)
    >>> result = model.fit()

    A mixed model with fixed effects for the columns of ``exog`` and
    independent random coefficients for the columns of ``exog_re``:

    >>> free = MixedLMParams.from_components(
                     fe_params=np.ones(exog.shape[1]),
                     cov_re=np.eye(exog_re.shape[1]))
    >>> model = sm.MixedLM(endog, exog, groups, exog_re=exog_re)
    >>> result = model.fit(free=free)

    A different way to specify independent random coefficients for the
    columns of ``exog_re``.  In this example ``groups`` must be a
    Pandas Series with compatible indexing with ``exog_re``, and
    ``exog_re`` has two columns.

    >>> g = pd.groupby(groups, by=groups).groups
    >>> vc = {}
    >>> vc['1'] = {k : exog_re.loc[g[k], 0] for k in g}
    >>> vc['2'] = {k : exog_re.loc[g[k], 1] for k in g}
    >>> model = sm.MixedLM(endog, exog, groups, vcomp=vc)
    >>> result = model.fit()
    NTnonec                 "
   g d}	|                                 D ]}
|
|	vrt          d|
z            || _        d| _        d | _        d | _        t          |t                    r#t          j	        d           t          |          }|!t          |j                  | _        || _        nd| _        t          g g g           | _        |,t!          j        |d           r|j        dk    r|d d d f         }|,t!          j        |d           r|j        dk    r|d d d f         } t'                      j        ||f|||d| | j                            dd	g           |j        d         | _        |t          | j        j                  dk    rd| _        d| _        t7          j        t          |          dft6          j        
          | _        | j        | j        _        dg}| j         |z   | j        _!        || j        _"        || j        _#        n|}|| j        _        t7          j$        |          | _        | j        j        dk    r| j        d d d f         | _        | j        j        d         | _        | j        | j        dz   z  dz  | _        nd| _        d| _        | j        j%        s=| &                    |          \  }}}|| j        _!        || j        _"        || j        _#        | j        | j        z   | _'        tQ          tS          |                    }|*                                 d |D             }tW          |          D ] \  }}||         ,                    |           !|| _-        || _.        t          | j.                  | _/        | 0                    | j1                  | _2        | 0                    | j3                  | _4        | 0                    | j                  | _5        | j        d | _6        nd | j5        D             | _6        t          | j1                  | _7        | j7        | _8        | j         .d ts          | j3        j        d                   D             | _         g | _:        g | _;        ts          | j/                  D ]`}| <                    |          }| j:        ,                    |           t{          |j>        |          }| j;        ,                    |           a| ?                                \  | _@        | _A        d S )N)missing_idxdesign_infoformulaz4argument %s not permitted for MixedLM initializationTz+Using deprecated variance components formatr   r   )r   rC   missingrZ   r   dtypez	Group VarrI   c                     i | ]}|g S r1   r1   )r7   ry   s     r   
<dictcomp>z$MixedLM.__init__.<locals>.<dictcomp>  s    333q"333r   c                 B    g | ]}t          j        |j        |          S r1   )r   r   r   r7   r   s     r   r9   z$MixedLM.__init__.<locals>.<listcomp>+  s$    HHH1qsAHHHr   c                     g | ]
}d |dz   z  S )zFE%dr   r1   r6   s     r   r9   z$MixedLM.__init__.<locals>.<listcomp>3  s-     : : :AvQ/ : : :r   )Br   
ValueErrorrZ   remlfe_penre_penr
   dictwarningswarnr   rR   r(   rL   r   r%   
data_tools_is_using_ndarray_typeru   superr,   
_init_keysextendr   rJ   r:   rK   r   onesfloat64rC   r   
exog_namesparam_namesexog_re_namesexog_re_names_fullasarray_param_names_make_param_namesk_paramsrA   r   r   	enumerater   row_indicesgroup_labelsn_groups
group_listendogendog_liexogexog_li
exog_re_liexog_re2_linobsn_totobsrB   _aex_r_aex_r2_augment_exogr   r   _reparam_lin_quad)r+   r   r   r   rC   r   rZ   r   kwargs_allowed_kwargsr   r(   r   r   r   r   r   r   r   ama	__class__s                        r   r,   zMixedLM.__init__  s    DCC 	P 	PA'' JQNP P P ( ! 	gt$$ 	+MGHHH!'**GGM**DI"DLLDI!"b"--DL 1$== 	Q4=D1'4@@  !!aaag&G 	 	#V!('	# 	#!	# 	# 	# 	
I6777 JqM	?s4<#566!;;DIDJ7CJJ?"*EEEDL $DI ME$(Oe$;DI!&+DI#+0DI((  !(DI:g..DL| A%%#|AAAtG4 *1-DIdi!m49DJJ DIDJy% 	> $(#9#9'#B#B![-$/DI!&3DI#+=DI(	DJ. CKK((33l333f%% 	% 	%DAqN!!!$$$$&(D-.. 
33ty11//$,77 <#DHHHHHD 
OO		 ?": :$TY_Q%788: : :DO t}%% 	$ 	$A""1%%AKq!!!ac1BL#### !%	4:::r   c                    t          | j                  }t          | |          }g }| j        }t	          t          |                    D ]k}t	          |dz             D ]V}||k    r|                    ||         dz              n*|                    ||         dz   ||         z   dz              |dz  }Wld | j        j        D             }||z   |z   ||fS )z
        Returns the full parameter names list, just the exogenous random
        effects variables, and the exogenous random effects variables with
        the interaction terms.
        r    Varz x z Covc                     g | ]}|d z   S )r   r1   r   s     r   r9   z-MixedLM._make_param_names.<locals>.<listcomp>W  s    ;;;1AJ;;;r   )	rA   r   rE   rJ   rB   rR   r   r   r(   )	r+   rC   r   r   r   jjr   r   r   s	            r   r   zMixedLM._make_param_namesC  s    $/**
*499Ys=))** 	 	A1q5\\  66&&}Q'7&'@AAAA&&}Q'7%'?'4Q'7(8:@(A B B Ba <;(:;;;K'(2M;NNr   Fc           	        ! d|	                                 vrt          d          |	d         }
d!t          |
t                    r|
!t	          j        ||
                   }
nt	          j        |
          }
|	d= |dk    rt          ||
|||          \  }}
d}||                                dk    r%t	          j        |j	        d         d	f          }!g}nw|	
                    d
d          }|d	}n|dk    rddlm}  |i           }t          j        |||          }|j        j        }!fd|D             }t	          j        |          }|j        d	k    r|dddf         }n
d}|!g}ng }||	
                    d
d          }|d	}n|dk    rddlm}  |i           }g }g }g }|                    |
          }t%          |j                                                   }t%          |                                           }|D ]1}t          j                            ||                   }|                    |           g g }}t/          |          D ]\  }}|j        |         }t          j        ||j        |ddf         |d          }|                    |j                                                   |r(|                    t7          j        |                     |                    t	          j        |                     |                    |           |                    |           3t;          |||          }nt;          g g g           }d|	d<   ||	d<   ||	d<   |
|	d<    t=                      j        ||g|R i |	}|                    |          \  }}} ||j         _!        ||j         _"        | |j         _#        ||j$        j%        |j         _&        |S )a$  
        Create a Model from a formula and dataframe.

        Parameters
        ----------
        formula : str or generic Formula object
            The formula specifying the model
        data : array_like
            The data for the model. See Notes.
        re_formula : str
            A one-sided formula defining the variance structure of the
            model.  The default gives a random intercept for each
            group.
        vc_formula : dict-like
            Formulas describing variance components.  `vc_formula[vc]` is
            the formula for the component with variance parameter named
            `vc`.  The formula is processed into a matrix, and the columns
            of this matrix are linearly combined with independent random
            coefficients having mean zero and a common variance.
        subset : array_like
            An array-like object of booleans, integers, or index
            values that indicate the subset of df to use in the
            model. Assumes df is a `pandas.DataFrame`
        missing : str
            Either 'none' or 'drop'
        args : extra arguments
            These are passed to the model
        kwargs : extra keyword arguments
            These are passed to the model with one exception. The
            ``eval_env`` keyword is passed to patsy. It can be either a
            :class:`patsy:patsy.EvalEnvironment` object or an integer
            indicating the depth of the namespace to use. For example, the
            default ``eval_env=0`` uses the calling namespace. If you wish
            to use a "clean" environment set ``eval_env=-1``.

        Returns
        -------
        model : Model instance

        Notes
        -----
        `data` must define __getitem__ with the keys in the formula
        terms args and kwargs are passed on to the model
        instantiation. E.g., a numpy structured or rec array, a
        dictionary, or a pandas DataFrame.

        If the variance component is intended to produce random
        intercepts for disjoint subsets of a group, specified by
        string labels or a categorical data value, always use '0 +' in
        the formula so that no overall intercept is included.

        If the variance components specify random slopes and you do
        not also want a random group-level intercept in the model,
        then use '0 +' in the formula to exclude the intercept.

        The variance components formulas are processed separately for
        each group.  If a variable is categorical the results will not
        be affected by whether the group labels are distinct or
        re-used over the top-level groups.

        Examples
        --------
        Suppose we have data from an educational study with students
        nested in classrooms nested in schools.  The students take a
        test, and we want to relate the test scores to the students'
        ages, while accounting for the effects of classrooms and
        schools.  The school will be the top-level group, and the
        classroom is a nested group that is specified as a variance
        component.  Note that the schools may have different number of
        classrooms, and the classroom labels may (but need not be)
        different across the schools.

        >>> vc = {'classroom': '0 + C(classroom)'}
        >>> MixedLM.from_formula('test_score ~ age', vc_formula=vc,                                   re_formula='1', groups='school', data=data)

        Now suppose we also have a previous test score called
        'pretest'.  If we want the relationship between pretest
        scores and the current test to vary by classroom, we can
        specify a random slope for the pretest score

        >>> vc = {'classroom': '0 + C(classroom)', 'pretest': '0 + pretest'}
        >>> MixedLM.from_formula('test_score ~ age + pretest', vc_formula=vc,                                   re_formula='1', groups='school', data=data)

        The following model is almost equivalent to the previous one,
        but here the classroom random intercept and pretest slope may
        be correlated.

        >>> vc = {'classroom': '0 + C(classroom)'}
        >>> MixedLM.from_formula('test_score ~ age + pretest', vc_formula=vc,                                   re_formula='1 + pretest', groups='school',                                   data=data)
        r   z?'groups' is a required keyword argument in MixedLM.from_formulaGroupdropr   N1r   r   eval_env)EvalEnvironment)r   c                 <    g | ]}|                     d           S )	Intercept)replace)r7   r   
group_names     r   r9   z(MixedLM.from_formula.<locals>.<listcomp>  s7     !9 !9 !9%& "#;
!C!C !9 !9 !9r   	dataframe)r   return_typesubsetrC   r   )'r   r   r
   r   r   r   _handle_missingstripr   r   getpatsyr   dmatrixr   column_namesru   groupbysortedr   	ModelDescfrom_formular   r   locr=   r>   r   
csr_matrixr%   r   r   r   r   r   r   r   r(   vcomp_names)#clsr   r   
re_formula
vc_formular   
use_sparser   argsr   r   rC   r   r   r   r   r   r   gbkylistvcfvc_namemdevc_matsevc_colnamesgroup_ixgroupr   matr   modr   r   r   r   s#                                    @r   r  zMixedLM.from_formula[  si   F 6;;==((  "; < < <! 
fc"" 	(JZV--FFZ''F8 f*4*+57 7LD&G!!!S(('4:a=!"455!+!::j$77# HH^^555555.r22H-
D8LLL ' 3 @!9 !9 !9 !9*7!9 !9 !9*W--|q  !!!!T'*G!!+ "!zz*d33HR111111*?2..GKHf%%BBINN,,--F**++C 1 1_11*W2EFF((()+R,'0'8'8 9 9OHe5)B-!Xb!!!e_&.)4	6 6 6C
 !''(:(:(<(<===! 9 (9#(>(>???? 
38888x(((""<0000X{G<<GGRR((Gx#y#y!x"egg"7DB4BBB6BB
  #44]CC			*!.&8#!#&;#4CH 
r   c                     || j         }t          |t                    r|j        }n|d| j                 }t          j        ||          S )a  
        Return predicted values from a design matrix.

        Parameters
        ----------
        params : array_like
            Parameters of a mixed linear model.  Can be either a
            MixedLMParams instance, or a vector containing the packed
            model parameters in which the fixed effects parameters are
            at the beginning of the vector, or a vector containing
            only the fixed effects parameters.
        exog : array_like, optional
            Design / exogenous data for the fixed effects. Model exog
            is used if None.

        Returns
        -------
        An array of fitted values.  Note that these predicted values
        only reflect the fixed effects mean structure of the model.
        Nr   )r   r
   rG   rT   rJ   r   r   )r+   rY   r   s      r   predictzMixedLM.predict&  sN    * <9Dfm,, 	)%FFAdiK(FvdF###r   c                 t     dS j         dk    r fd j        D             S  fd j        D             S )zg
        Returns `array` split into subarrays corresponding to the
        grouping structure.
        Nr   c                 Z    g | ]'}t          j        j        |                            (S r1   r   rX   r   r7   r8   rX   r+   s     r   r9   z&MixedLM.group_list.<locals>.<listcomp>O  s@     0 0 0 HU4#3A#6788 0 0 0r   c                 b    g | ]+}t          j        j        |         d d f                   ,S r'   r   r!  s     r   r9   z&MixedLM.group_list.<locals>.<listcomp>R  sI     0 0 0 HU4#3A#6#9:;; 0 0 0r   )ru   r   )r+   rX   s   ``r   r   zMixedLM.group_listE  s     =4:??0 0 0 0 0!.0 0 0 00 0 0 0 0!.0 0 0 0r   l1r   -C6?ư>   c                 b	   t          |t                    r'|                                dk    rt          d          t          |t                    r+||_        |                    d|i            | j        di |S t          j	        |          r(|t          j
        | j        t          j                  z  } | j        di |}|j        }	|j        }
|j        }|j        }	 t          j                            |
          }n# t          j        j        $ r d}Y nw xY wt)          |          D ]}|	                                }t)          | j                  D ]J}t-          |	|                   |k     rd|	|<   t          j        | j        |	          }| j        |z
  }d\  }}t5          | j                  D ]\  }}|                     ||          }| j        |         }| j        |         | j        |         }}|| j         |                  }tC          ||||d|z            }|dd|f         } ||          }|t          j        ||          z  }|d	t          j        ||          z  z  }||         }||k    r||z
   d	|z  z  |	|<   4|| k     r||z    d	|z  z  |	|<   Lt          j        ||	z
            "                                |k     r n|j#                                        } |	| d
| j        <   | $                    |	|j%        |j                  }| &                    |           \  }!}"|"rtO          j(        tR                     t          j*        t          j+        |!          z  }#t          j        |           |k    }$d|$| j        d<   t          j,        |$          }$|!|$ddf         dd|$f         }%t          j                            |%           |#t          j-        |$|$          <   t\          /                    |	|
          }&ta          | | |#|z            }'|&|'_1        |	|'_        |
|'_        ||'_        ||'_        |j%        |'_%        |j2        |'_2        d|'_3        | j4        |'_4        | j        |'_        | j5        |'_5        | j6        |'_6        | j7        |'_7        tq          |'          S )a  
        Fit a model in which the fixed effects parameters are
        penalized.  The dependence parameters are held fixed at their
        estimated values in the unpenalized model.

        Parameters
        ----------
        method : str of Penalty object
            Method for regularization.  If a string, must be 'l1'.
        alpha : array_like
            Scalar or vector of penalty weights.  If a scalar, the
            same weight is applied to all coefficients; if a vector,
            it contains a weight for each coefficient.  If method is a
            Penalty object, the weights are scaled by alpha.  For L1
            regularization, the weights are used directly.
        ceps : positive real scalar
            Fixed effects parameters smaller than this value
            in magnitude are treated as being zero.
        ptol : positive real scalar
            Convergence occurs when the sup norm difference
            between successive values of `fe_params` is less than
            `ptol`.
        maxit : int
            The maximum number of iterations.
        **fit_kwargs
            Additional keyword arguments passed to fit.

        Returns
        -------
        A MixedLMResults instance containing the results.

        Notes
        -----
        The covariance structure is not updated as the fixed effects
        parameters are varied.

        The algorithm used here for L1 regularization is a"shooting"
        or cyclic coordinate descent algorithm.

        If method is 'l1', then `fe_pen` and `cov_pen` are used to
        obtain the covariance structure, but are ignored during the
        L1-penalized fitting.

        References
        ----------
        Friedman, J. H., Hastie, T. and Tibshirani, R. Regularized
        Paths for Generalized Linear Models via Coordinate
        Descent. Journal of Statistical Software, 33(1) (2008)
        http://www.jstatsoft.org/v33/i01/paper

        http://statweb.stanford.edu/~tibs/stat315a/Supplements/fuse.pdf
        r#  zInvalid regularization methodr   r   N        r(  r(  r   rI   r   T)rV   r1   )9r
   r   lowerr   r   alphaupdatefitr   isscalarr   rJ   r   rT   rV   rW   scalerh   invrj   rB   re   absr   r   r   r   r   _expand_vcompr   r   r   r   r   maxrY   	get_scalecov_re_unscaledhessianr   r   _warn_cov_singnan	ones_likeflatnonzeroix_rG   rc   MixedLMResultsparams_objectmethod	convergedcov_penr:   rK   rL   MixedLMResultsWrapper)(r+   start_paramsr>  r+  cepsptolmaxit
fit_kwargsmdfrT   rV   rW   r/  
cov_re_invitrfe_params_sr   expval	resid_allr   br  r  vc_varr   ex_rex2_rresidrz   r   upwt1params_profhesssingpcovr   hess1r=  resultss(                                           r   fit_regularizedzMixedLM.fit_regularizedU  s   n fc"" 	>$(>(><=== fg&& 	* FLx011148))j)));u 	ABGDIRZ@@@@E dh$$$$M				v..JJy$ 	 	 	JJJ	 << '	 '	C#..**K49%% !9 !9y|$$t++  "	!	955 J/	 1'01B'C'C . .OHe!//x@@F<1D"&+h"7h9O%D%d&6u&=>E(eZ)*V5 5F QQQT
Aq		A1%ARVAu----AAQxt88%&X;!a%#8IaLL$YY%&X;!a%#8IaLvkI-..2244t;; < joo''#,AdiK y#*=syII \\+..
d 	*M.)))vT***VK  4'49::^BRUAAArE"!y}}eV44RVB^^%55i5OO {D5LAA -%"%"5 ,yy
y$W---s   #D DD绽|=c                 h   | j         dk    rt          j        g           dfS d}| j        dk    rt          j        d          }nt          j                            |          \  }}|                                |k     rtd}t          j        ||k              }t          |          dk    rt          j
        |          }nQ|dd|f         }	||         }
t          j        |	|
z  |	j                  }nt          j                            |          }t          | d          ssg | _        t!          | j                  D ]W\  }}t          j        | j        |         | j        |         dddf         fd          }| j                            |           Xd	}t!          | j                  D ]\  }}|                     ||          }|j        dk    r[|                                |k     r=d}t          j        ||k              }t          j
        |          }d||         z  ||<   nd|z  }nt          j        d          }| j        |         }| j        |         | j        |         }}t5          d
||||          } || j        |                   }|t          j        |j        |          z  }|rIt          j        t          j                            |ddddf                   |dddf                   }n6t          j                            |ddddf         |dddf                   }||fS )a>  
        Use GLS to update the fixed effects parameter estimates.

        Parameters
        ----------
        cov_re : array_like (2d)
            The covariance matrix of the random effects.
        vcomp : array_like (1d)
            The variance components.
        tol : float
            A tolerance parameter to determine when covariances
            are singular.

        Returns
        -------
        params : ndarray
            The GLS estimates of the fixed effects parameters.
        singular : bool
            True if the covariance is singular
        r   Fr`   TN	_endex_lir   axisr(        ?r   )rJ   r   rX   r:   ra   rh   eighminr:  rR   
zeros_liker   r   r0  hasattrr]  r   r   rl   r   r   r   r2  sizer   r   r   pinvr   )r+   rV   rW   tolrV  rH  wr   r   viwir  r   r  xtxyr  rN  vc_varir   rO  rP  rz   rR  rT   s                           r   get_fe_paramszMixedLM.get_fe_params  s   , 9>>8B<<&&9>>&))JJ9>>&))DAquuww}}^AH--r77a<<!#v!6!6JJ111b5B2B!#R!6!6JJY]]622
 t[)) 	+DN():;; + +!n\(+]8,QQQW57=>@ @ @ %%c****():;; 	& 	&OHe''x88F{Q::<<#%%D#66B mF33G"#fRj.GBKK&jGG(1++<)D+h/h1G%D T5*gFFFt~h/00ABF461%%%DD 	Dry~~d111ad7m<<d111b5kJJII	QQQ"WtAAArE{CCI$r   c                    | j         | j        | j        | j        f\  }}}}||z   |z   }t	          j        | j                  }g }t          |          D ]0}t	          j        |          }	d|	|<   |                    |	           1t          |          D ])}|                    t	          j        |                     *t          |          D ])}|                    t	          j        |                     *g }
t          |          D ]+}|
                    t	          j        ||f                     ,t	          j        |          }d t          |d         |d                   D             }t          |          D ]}t          |          D ]|}||         }||         }|d         |d         k    rX|d         |d         k    rF|d         |d         f}|
                    |          }|
||z            ||z   ||z   fxx         dz  cc<   }t          |          D ]}d|
|         |
|         j        z   z  |
|<   ||z   }t          |||z             D ]}d|
|         ||f<   ||
fS )aB  
        Returns parameters of the map converting parameters from the
        form used in optimization to the form returned to the user.

        Returns
        -------
        lin : list-like
            Linear terms of the map
        quad : list-like
            Quadratic terms of the map

        Notes
        -----
        If P are the standard form parameters and R are the
        transformed parameters (i.e. with the Cholesky square root
        covariance and square root transformed variance components),
        then P[i] = lin[i] * R + R' * quad[i] * R
        r   c                     g | ]	\  }}||f
S r1   r1   )r7   r   rM  s      r   r9   z$MixedLM._reparam.<locals>.<listcomp>n  s     333Aq!f333r   r         ?)rJ   r:   rK   rL   r   rN   rB   rS   r   zipindexr   )r+   rJ   r:   rK   rL   rM   r]   linr8   equadr   i1i2ix1ix2kms                    r   r   zMixedLM._reparamE  s   ( #')TY
DI"MdE4ut#_TY''t 	 	AAAaDJJqMMMMu 	( 	(AJJrx''''t 	( 	(AJJrx''''u 	2 	2AKK%001111 _T""33RUBqE!2!2333,, 	8 	8BEll 8 8ffFc!f$$3q6SV+;+;a&#a&)BAaLb$r'!1222a72228 u 	0 	0A47T!WY./DGG E\r2d7## 	 	ADGAqDMMDyr   c                    t          |          dk    rt          j        d          S g }t          t          | j        j                            D ]U}| j        j        |         |         j        d         }|                    ||         t          j	        |          z             Vt          |          dk    rt          j
        |          S t          j        d          S )a  
        Replicate variance parameters to match a group's design.

        Parameters
        ----------
        vcomp : array_like
            The variance parameters for the variance components.
        group_ix : int
            The group index

        Returns an expanded version of vcomp, in which each variance
        parameter is copied as many times as there are independent
        realizations of the variance component in the given group.
        r   r   )rR   r   ra   rB   r   r(   r*   r   r   r   rl   )r+   rW   r  rN  r   r   s         r   r2  zMixedLM._expand_vcomp  s     u::??8A;;s4<-..// 	1 	1A!!$X.4Q7AMM%(RWQZZ/0000v;;??>&))) 8A;;r   c                 H   | j         dk    r| j        |         nd}| j        dk    r|S | j         dk    r|gng }d}t          | j        j                  D ]M\  }}|                    | j        j        |         |                    |t          j	        |d                   z  }N|rit          |          D ]0\  }}t          j	        |          st          j
        |          ||<   1t          j        |          }t          j
        |          }nt          j        |d          }|S )z
        Concatenate the columns for variance components to the columns
        for other random effects to obtain a single random effects
        exog matrix for a given group.
        r   NFr   r   r^  )r:   r   rL   r   r   r(   r   r*   r   r   r
  hstackr   rl   )r+   r  rO  ex
any_sparser   r   r   s           r   r   zMixedLM._augment_exog  s&    -1IMMtx((t9>>Ky1}}dVV"
dl011 	2 	2DAqIIdl'*84555&/"R&111JJ 	,!" 1 11q)) 1"-a00BqEr""B"2&&BB+++B	r   c           	      n   t          |          t          ur.t                              || j        | j        | j        d          }|j        }|j        }|r,|                     ||          \  }}|r| xj	        dz  c_	        n|j
        }| j        dk    r	 t          j                            |          }nI# t          j        j        $ r2 t          j                            |          }| xj	        dz  c_	        Y nw xY wt          j                            |          \  }}	nt          j        d          }d}	t          j        | j        |          }
| j        |
z
  }d}| j        )| j        dk    r|| j                            ||          z  }| j        || j                            |          z  }d\  }}t1          | j                  D ]\  }}|                     ||          }|	t          j        t          j        |                    z   }| j        |         }| j        |         | j        |         }}tA          d	|||d|z            }|| j!        |                  }tE          d	|||d|z  |          }||d
z  z  } ||          }|t          j        ||          z  }| j#        r( ||          }|t          j        |j$        |          z  }| j#        r|| j%        | j        z
  t          j        |          z  d
z  z  }t          j                            |          \  }}||d
z  z  }|| j%        | j        z
  t          j        dt          j&        z            z  d
z  z  }|| j%        | j        z
  t          j        | j%        | j        z
            z  d
z  z  }|| j%        | j        z
  d
z  z  }n|| j%        t          j        |          z  d
z  z  }|| j%        t          j        dt          j&        z            z  d
z  z  }|| j%        t          j        | j%                  z  d
z  z  }|| j%        d
z  z  }|S )a  
        Evaluate the (profile) log-likelihood of the linear mixed
        effects model.

        Parameters
        ----------
        params : MixedLMParams, or array_like.
            The parameter value.  If array-like, must be a packed
            parameter vector containing only the covariance
            parameters.
        profile_fe : bool
            If True, replace the provided value of `fe_params` with
            the GLS estimates.

        Returns
        -------
        The log-likelihood value at `params`.

        Notes
        -----
        The scale parameter `scale` is always profiled out of the
        log-likelihood.  In addition, if `profile_fe` is true the
        fixed effects parameters are also profiled out.
        Fr[   r   r   r`   r(  Nr)  r`  g       @rI   )'typerG   r^   rJ   r:   rZ   rV   rW   rm  	_cov_singrT   r   rh   r0  rj   rf  r   rS   r   r   r   r@  funcr   r   r   r2  r!   r   r   r   r   r   r   r   r   r   r   pi)r+   rY   
profile_ferV   rW   rT   rV  rH  r   cov_re_logdetrK  rL  likevalxvxqfr  r  rN  cov_aug_logdetr   rO  rP  rz   rQ  r   rR  r  s                              r   loglikezMixedLM.loglike  s,   4 <<},,"..vty/3y$-6; / = =F   	)"00??OIt $!#(I9q==$Y]]622

9( $ $ $Y^^F33
!#$  "y0088A}}&))JM 	9--J'	 L$49q==t|((<<<G K#t{''	222GR():;; 	+ 	+OHe''x88F*RVBF6NN-C-CCN<)D+h/h1G%D T5*a&jIIFd.u56E Ruj!f*+- -BrBwG uA"&"""B y +fTllrvdfc***9 	*	1RVBZZ?"DDGI%%c**EArrBwG	1RVAI5F5FFKKG2t}ty899:<>? @G	1R77GGt}rvbzz1B66Gt}rva"%i'8'882==Gt}rvdm'<'<<rAAGt}r))Gs   B4 4AC:9C:c           	   #     K    ||          }d}t          | j                  D ]z}t          |dz             D ]e}|
||k    r  dS |dd||dz   f         |dd||dz   f         }
}	|dd||dz   f         |dd||dz   f         }}||	|
||||k    fV  |dz  }f{t          | j        j                  D ]B\  }}|	||k    r dS | j        j        |         |         } ||          }|||||dfV  |dz  }CdS )a8  
        A generator that yields the element-wise derivative of the
        marginal covariance matrix with respect to the random effects
        variance and covariance parameters.

        ex_r : array_like
            The random effects design matrix
        solver : function
            A function that given x returns V^{-1}x, where V
            is the group's marginal covariance matrix.
        group_ix : int
            The group index
        max_ix : {int, None}
            If not None, the generator ends when this index
            is reached.
        r   r   NT)rB   r:   r   r   r(   r*   )r+   rO  rz   r  max_ixaxrr   j1j2mat_lmat_rvslvsrr   r   r  axmats                    r   _gen_dV_dParzMixedLM._gen_dV_dPar#  s|     $ fTll 	"" 	 	BBFmm  %"v++FFF#AAAr"Q$wJ/aaaBqDj1Auqqq"RT'z?C2bd7
OS%S"(::::a dl011 	 	DAq!b6kk,#A&x0CF3KKEc3ud2222!GBB	 	r   c                 t   t          |          t          ur.t                              || j        | j        | j        d          }|r@|                     |j        |j                  \  |_	        }|rd}t          j        |           | j        r|                     ||           \  }}}n|                     ||           \  }}}| j        =|| j        j	        z  }|| j        j        | j        j                 z  }|| j        j        z  }|rt!          j        ||f          S t!          j        |||f          S )a  
        Returns the score vector of the profile log-likelihood.

        Notes
        -----
        The score vector that is returned is computed with respect to
        the parameterization defined by this model instance's
        `use_sqrt` attribute.
        Fr  z%Random effects covariance is singularcalc_fe)r  rG   r^   rJ   r:   rZ   rm  rV   rW   rT   r   r   
score_sqrt
score_full_freepatrO   r   rl   )r+   rY   r  rV  msgscore_fescore_rescore_vcs           r   scorezMixedLM.scoreL  s]    <<},,"..	49dm /  F  	#""6=&,?? #Fd  #=c"""= 	0+/??J ,; ,0 ,0(Hh ,0??J ,; ,0 ,0(Hh =$//H,T]->??H++H 	B>8X"6777>8Xx"@AAAr   c                 \
   |j         }|j        }|j        }	 t          j                            |          }nI# t          j        j        $ r2 t          j                            |          }| xj        dz  c_        Y nw xY wt          j	        | j
                  }t          j	        | j                  }t          j	        | j                  }	| j        || j                            ||          z  }|r$| j        || j                            |          z  }d}
d}d}dg| j        | j        z   z  }t          j	        | j        | j        z             }t          j	        | j        | j        z             }t!          | j                  D ]r\  }}|                     ||          }| j        |         }| j        |         | j        |         }}t-          d|||d|z            }| j        |         }| j
        dk    rt          j        ||          }||z
  }| j        r( ||          }|t          j        |j        |          z  } ||          }|                     |||          D ]\  }}}}}} t9          ||          ||<   | s||xx         t9          ||          z  cc<   t;          ||          }!| r|!j        nt;          |j        |          }"t          j        |!|"          }#||xx         |#z  cc<   | s||xx         |#j        z  cc<   | j        rot;          |j        |          }!| r|!j        nt;          |j        |          }"t          j        |!|"          }#||xx         |#z  cc<   | s||xx         |#j        z  cc<   | j        dk    r|d|d| j                 z  z  }| j        dk    r|	d|| j        d         z  z  }	|
t          j        ||          z  }
|r|t          j        |j        |          z  }t| j        }$| j        r
|$| j
        z  }$|r| j
        dk    r||$|z  |
z  z  }| j        dk    r|d|$z  |d| j                 z  |
z  z  }| j        dk    r|	d|$z  || j        d         z  |
z  z  }	| j        rt          j                            |          }%tA          | j                  D ].}&||&xx         dt9          |%j        ||&                   z  z  cc<   /tA          | j                  D ]6}&|	|&xx         dt9          |%j        || j        |&z                      z  z  cc<   7|||	fS )a  
        Returns the score with respect to untransformed parameters.

        Calculates the score vector for the profiled log-likelihood of
        the mixed effects model with respect to the parameterization
        in which the random effects covariance matrix is represented
        in its full form (not using the Cholesky factor).

        Parameters
        ----------
        params : MixedLMParams or array_like
            The parameter at which the score function is evaluated.
            If array-like, must contain the packed random effects
            parameters (cov_re and vcomp) without fe_params.
        calc_fe : bool
            If True, calculate the score vector for the fixed effects
            parameters.  If False, this vector is not calculated, and
            a vector of zeros is returned in its place.

        Returns
        -------
        score_fe : array_like
            The score vector with respect to the fixed effects
            parameters.
        score_re : array_like
            The score vector with respect to the random effects
            parameters (excluding variance components parameters).
        score_vc : array_like
            The score vector with respect to variance components
            parameters.

        Notes
        -----
        `score_re` is taken with respect to the parameterization in
        which `cov_re` is represented through its lower triangle
        (without taking the Cholesky square root).
        r   Nr(  r`  r   rp  )!rT   rV   rW   r   rh   r0  rj   rf  r  rS   rJ   rK   rL   r@  derivr   r   r   r2  r   r   r   r   r   r   r   r   r  r#   r   r:   r   rB   )'r+   rY   r  rT   rV   rW   rH  r  r  r  rvirxtvirxtvixxtaxdlvrvavrr  r  rN  r   rO  rP  rz   rQ  rK  viexogvirr   matlmatrr  r  symulurulrfacxtvixir   s'                                          r   r  zMixedLM.score_fullu  sx   N $		 v..JJy$ 	  	  	 //JNNaNNNN	  8DI&&8DJ''8DI&& <#**6:>>>H  	5/)))444H    vdi/0 htzDI-.. di/00():;; 3	- 3	-OHe''x88F<)D+h/h1G%D T5*a&jIIF M(+Ey1}}i00y 0/// &--C%%dFH==* */T4c3!$,,B 2GGGwtS111GGG#t__ 7RTTd463&7&7fRnnb			S 			 '"III&III9 *fh--B!$>$tvv*>*>B&R..CHHHOHHH *RCE) y1}}C#a
l"333y1}}C#djkk"222BF5#&&&D -,,,m9 	49C 	+ty1}}ed**H9q==c	E!DJ,$77$>>H9q==c	E$*++$66==H9 	MY]]5))F4:&& @ @sWVXtAw%?%???49%% M MsWVXtDJN7K%L%LLL8++s   7 AA=<A=c                    |                      ||          \  }}}|                    dd          }t          j        |||f          }d}t	          t          |                    D ]A}	| j        |	         dt          j        | j        |	         |          z  z   }
|||	         |
z  z  }B|d| j	                 }|| j	        | j	        | j
        z            }|| j	        | j
        z   d         }|||fS )ak  
        Returns the score with respect to transformed parameters.

        Calculates the score vector with respect to the
        parameterization in which the random effects covariance matrix
        is represented through its Cholesky square root.

        Parameters
        ----------
        params : MixedLMParams or array_like
            The model parameters.  If array-like must contain packed
            parameters that are compatible with this model instance.
        calc_fe : bool
            If True, calculate the score vector for the fixed effects
            parameters.  If False, this vector is not calculated, and
            a vector of zeros is returned in its place.

        Returns
        -------
        score_fe : array_like
            The score vector with respect to the fixed effects
            parameters.
        score_re : array_like
            The score vector with respect to the random effects
            parameters (excluding variance components parameters).
        score_vc : array_like
            The score vector with respect to variance components
            parameters.
        r  TrZ   r[   r(  rI   r   N)r  ro   r   rl   rB   rR   r   r   r   rJ   rK   )r+   rY   r  r  r  r  
params_vecr  scrr   r   s              r   r  zMixedLM.score_sqrt  s    > (,vw'O'O$(H&&T&BB
^Xx$BCC
s:'' 	% 	%A	!q26$*Q-#D#DDDA:a=1$$CCq{#tyTZ!778ty4:-../8++r   c                   789:; t          |          t          ur.t                              || j        | j        | j        d          }|j        }|j        }|j        }d}| j        dk    r\	 t          j
                            |          }nO# t          j
        j        $ r$ t          j
                            |          }d}Y nw xY wt          j        d          }d}t          j        | j        | j        z   | j        | j        z   f          }t          j        | j        | j        z   | j        f          }	| j        }
| j        r|
| j        j        d         z  }
d}d;dg| j        | j        z   z  }| j        | j        z   7t          j        7          }t          j        77f          }7fdt-          7          D             }t/          | j                  D ]K\  }}|                     ||          }t          j        |          }t          j        |d	k              }t9          |          dk    rd||         z  ||<   t9          |          t9          |          k     rd}| j        |         }| j        |         | j        |         }}tA          d
||||          }| j!        |         }| j        dk    rt          j"        ||          }||z
  } ||          :;t          j"        |j#        :          z  ; ||          }|t          j"        ||          z  }| $                    |||          D ]\  }}}}} }!tK          :j#        |          }"tK          |j#        |          }#|	|ddfxx         t          j"        |"|#          z  cc<   |!sQtK          :j#        |          }"tK          |j#        |          }#|	|ddfxx         t          j"        |"|#          z  cc<   | j        rotK          :j#        |          }"|!r|"nt          j"        :j#        |          }#tK          |"|#j#                  }$||xx         |$z  cc<   |!s||xx         |$j#        z  cc<   tK          ||          }"|!r|"ntK          ||          }#||xx         t          j"        |"|#          |!rdndz  z  cc<   ||fg}%|!s|%&                    | |f           | $                    ||||          D ]F\  }&89}'}(})tO          9fd|%D                       }*dtK          tQ          |dddf         8|*          |dddf                   z  }+|)sTtO          8fd|%D                       },|+dtK          tQ          |dddf         9|,          |dddf                   z  z  }+|||&fxx         t          j)        |+          z  cc<   ||&k    r$||&|fxx         t          j)        |+          z  cc<   tU          |'|*j#                  dz  }-|)s|-tU          |(|,j#                  dz  z  }-|||&fxx         |-z  cc<   ||&k    r||&|fxx         |-z  cc<   | j        rtO          :fd|%D                       }.tK          :j#        8          }/tK          9j#        |.          }0t          j"        |/|0          }1||         |&xx         |1|1j#        z   z  cc<   |)sgt          j"        :j#        9          }/t          j"        8j#        |.          }0t          j"        |/|0          }1||         |&xx         |1|1j#        z   z  cc<   HM||
;z  |z  z  }|d|
z  ||z  t          j+        ||          |dz  z  z
  z  z
  }|
 |	z  |z  }	| j        rӈ;fd|D             }2t-          | j        | j        z             D ]}3t-          |3dz             D ]}4tU          |2|3         j#        |2|4                   }5|5t          j,        t          j
        -                    ;||3         |4                             z  }5|5dz  }5||3|4fxx         |5z  cc<   |3|4k    r||4|3fxx         |5z  cc<   | j        | j        z   | j        z   7t          j        77f          }6||6d| j        d| j        f<   |	j#        |6d| j        | j        df<   |	|6| j        dd| j        f<   ||6| j        d| j        df<   |6|fS )a!  
        Returns the model's Hessian matrix.

        Calculates the Hessian matrix for the linear mixed effects
        model with respect to the parameterization in which the
        covariance matrix is represented directly (without square-root
        transformation).

        Parameters
        ----------
        params : MixedLMParams or array_like
            The model parameters at which the Hessian is calculated.
            If array-like, must contain the packed parameters in a
            form that is compatible with this model instance.

        Returns
        -------
        hess : 2d ndarray
            The Hessian matrix, evaluated at `params`.
        sing : boolean
            If True, the covariance matrix is singular and a
            pseudo-inverse is returned.
        Tr  Fr   r`   r(  r   c                     g | ]}d gz  	S )r(  r1   )r7   r8   r   s     r   r9   z#MixedLM.hessian.<locals>.<listcomp>y  s    (((!bTAX(((r   r[  r`  NrI   c                 ^    g | ])}t          j        |d          |d         j                  *S r   r   r   r   )r7   r   matr2s     r   r9   z#MixedLM.hessian.<locals>.<listcomp>  sA     * * *"# /uw!adfEE * * *r   c                 ^    g | ])}t          j        |d          |d         j                  *S r  r  )r7   r   matl2s     r   r9   z#MixedLM.hessian.<locals>.<listcomp>  sA     ". ". ".&' #357AaD!A$&"I"I ". ". ".r   c           	      n    g | ]1}t          |d          t          |d         j                            2S r  )r   r   )r7   r   r  s     r   r9   z#MixedLM.hessian.<locals>.<listcomp>  s7    !N!N!Nq$qtT!A$&&-A-A"B"B!N!N!Nr   rp  c                 P    g | ]"}t           j                            |          #S r1   )r   rh   r   )r7   r   r  s     r   r9   z#MixedLM.hessian.<locals>.<listcomp>  s)    :::")//%++:::r   ).r  rG   r^   rJ   r:   rZ   rT   rW   rV   r   rh   r0  rj   rf  ra   rS   rK   rL   r   r   r   r   rB   r   r   r2  rc  r:  rR   r   r   r   r   r   r   r   r  r   r   r!   r   squeezer#   outertracer   )<r+   rY   rT   rW   rV   rV  rH  hess_fehess_re	hess_ferer  r  r  r   DFr  r  rN  rl  r   r   rO  rP  rz   rQ  rK  r  jj1matl1matr1vsl1vsr1sym1r  r  r  Ejj2vsl2vsr2sym2revtlertevu1u2umQLr  r  r   rU  r   r  r  r  r  s<                                                          @@@@@r   r6  zMixedLM.hessian>  s	   2 <<},,"..vty$)8<6: / < <F $	9q==Y]]622

9(   Y^^F33
 &))J (DJ2DJ4JKLLHdj494di@AA	m9 	&49?1%%Cvdi/0J"HQKKHaV((((uQxx(((():;; Y	5 Y	5OHe''x88FmF++G%00B2ww{{&*n2wwV$$<)D+h/h1G%D T5*gFFF M(+Ey1}}i00VD\\FRVDFF+++E&--CBF5#&&&D %%dFH==@5 @55eUD$ &(E**%'3''#qqq&!!!RVB^^3!!! 8fh..Begs++Bc111f%%%B7%%%9 +fh..B#@%)@)@Br24..CIII$III +S			SU*			#u%%5RRT#u%5%5#"&R..,<AA1== E]O ,HHdE]+++ ))$#FF$5 $59S%dD  * * * *'(* * * + +BT"23tQQQw<"K"K"%aaag,0 0 0B   D  ". ". ". ".+,". ". ". / /a$'7aaaL%(5 (569!!!T'l#D #D D D c3hKKK2:b>>1KKKczz#s(rz"~~5 rt,,q0B 6gdBD11A55CH%%%+%%%czzS)))R/)))y 
5 !N!N!N!NA!N!N!NOO!&(E22!%'2..VB^^#srBDy0# 5!#%!8!8B!#!4!4B!#BBcF3KKK294KKKI$59@5D 	3;%%C#I4"(1a..472J)JKKD9$t+	9 		-::::T:::BDJ233 - -Q-- - -B2"R&11A")//%2r"C"CDDDAHABFOOOq(OOOBwwB1,- I
"TY.xA)0Qty[!DI+%&(1Qty[$)**$%(1TYZZ49$%'.TYZZ#$Tzs   -B 5CCc           	         	 t           j                            |          }nR# t           j        j        $ r; t           j                            |          }t          j        t                     Y nw xY wd}t          | j	                  D ]\  }}| 
                    ||          }| j        |         }	| j        |         | j        |         }}
t          d|
||d|z            }| j        |         }| j        dk    rt          j        |	|          }||z
  } ||          }|t          j        ||          z  }| j        r|| j        | j        z
  z  }n
|| j        z  }|S )a  
        Returns the estimated error variance based on given estimates
        of the slopes and random effects covariance matrix.

        Parameters
        ----------
        fe_params : array_like
            The regression slope estimates
        cov_re : 2d array_like
            Estimate of the random effects covariance matrix
        vcomp : array_like
            Estimate of the variance components

        Returns
        -------
        scale : float
            The estimated error variance.
        r(  r`  r   r   )r   rh   r0  rj   rf  r   r   r7  r   r   r2  r   r   r   r   r   rJ   r   r   r   )r+   rT   rV   rW   rH  r  r  r  rN  r   rO  rP  rz   rQ  rK  r  s                   r   r4  zMixedLM.get_scale  sc   (	*v..JJy$ 	* 	* 	*//JM.)))))	* ():;; 	% 	%OHe''x88F<)D+h/h1G%D T5*a&jIIF M(+Ey1}}i00&--C"&$$$BB9 	 4=49,-BB$-B	s   " AA10A1c
                    g d}|
                                 D ]}||vrt          j        d|z             |	g d}	nt          |	t                    r|	g}	|	D ]*}|                                dv rt          d|z            +|| _        || _        || _	        d| _
        || _        |rg }nd}|{t          | j        | j        | j                  }t!          j        | j                  |_        t!          j        | j                  |_        t!          j        | j                  |_        nt          |t                    r|}nt/          |          | j        | j        z   | j        z   k    r/t                              || j        | j        | j        d	          }n^t/          |          | j        | j        z   k    r/t                              || j        | j        | j        d
	          }nt          d          |r.|du|
d<   d|
vrd
|
d<   |                    | j        d
          }|dk    rt          j        d           t9          t/          |	                    D ]} t;                      j        d|d|	|         d|
}|j        d         r nc|j         }|dz   t/          |	          k     r)|	|dz            }t          j        d|z  tB                     zd}t          j        |tB                     t!          j"        |j                   }||#                    |j                   |j        d         }|sb| $                    |j                   }t!          j%        t!          j&        |dz                      }d|z  }t          j        |tB                     t                              || j        | j        | j        d
          }|j        }|j        }| '                    ||          \  }}||_        | (                    |||          }||z  }||z  }| j        dk    o;t!          j)        t!          j*        t!          j+        |                              dk     }| j        dk    o)t!          j)        t!          j*        |                    dk     }|s|rd}t          j        |tB                     | ,                    |          \  } }|rt          j        tZ                     t!          j+        |           }!|t!          j.        |           }"| j                            d
d          }#t!          j/        |#          }$|!|$         }!t/          |$          dk    rQ| t!          j0        |$|$                   }%t           j1        2                    |%           |"t!          j0        |$|$          <   n t           j1        2                    |            }"t!          j3        |!dk              rd}t          j        |tB                     |                    d
d          }&ti          | |&|"|z            }'||'_5        ||'_        ||'_        ||'_        ||'_6        ||'_7        | j        rdnd|'_8        ||'_9        ||'_:        | j        |'_        | j        |'_        | j        |'_        | j        |'_        | j        |'_        | j        |'_        | j        |'_        | j        |'_;        ty          |'          S )a  
        Fit a linear mixed model to the data.

        Parameters
        ----------
        start_params : array_like or MixedLMParams
            Starting values for the profile log-likelihood.  If not a
            `MixedLMParams` instance, this should be an array
            containing the packed parameters for the profile
            log-likelihood, including the fixed effects
            parameters.
        reml : bool
            If true, fit according to the REML likelihood, else
            fit the standard likelihood using ML.
        niter_sa : int
            Currently this argument is ignored and has no effect
            on the results.
        cov_pen : CovariancePenalty object
            A penalty for the random effects covariance matrix
        do_cg : bool, defaults to True
            If False, the optimization is skipped and a results
            object at the given (or default) starting values is
            returned.
        fe_pen : Penalty object
            A penalty on the fixed effects
        free : MixedLMParams object
            If not `None`, this is a mask that allows parameters to be
            held fixed at specified values.  A 1 indicates that the
            corresponding parameter is estimated, a 0 indicates that
            it is fixed at its starting value.  Setting the `cov_re`
            component to the identity matrix fits a model with
            independent random effects.  Note that some optimization
            methods do not respect this constraint (bfgs and lbfgs both
            work).
        full_output : bool
            If true, attach iteration history to results
        method : str
            Optimization method.  Can be a scipy.optimize method name,
            or a list of such names to be tried in sequence.
        **fit_kwargs
            Additional keyword arguments passed to fit.

        Returns
        -------
        A MixedLMResults instance.
        )gtolmaxiterepsmaxcorftolrg  dispmaxlsz#Argument %s not used by MixedLM.fitN)bfgslbfgscg)newtonncgz#method %s not available for MixedLMr   Tr  Fzinvalid start_paramsretallr  r  zniter_sa is currently ignored)rB  skip_hessianr>  r?  r   z%Retrying MixedLM optimization with %szCMixedLM optimization failed, trying a different optimizer may help.rI   z)Gradient optimization failed, |grad| = %fg{Gz?z6The MLE may be on the boundary of the parameter space.zNThe Hessian matrix at the estimated parameter values is not positive definite.REMLMLr1   )=r   r   r   r
   r   r*  r   r   r@  r   r  r  rG   rJ   r:   rL   r   rS   rT   eyerV   r   rW   rR   rK   r^   rZ   ro   rB   r   r-  mle_retvalsrY   r   
atleast_1dr   r  rk   r!   rm  r4  rb  r1  rU   r6  r7  rc  r:  r;  rh   r0  anyr<  r=  r/  r5  r>  r?  histfreepatrA  ))r+   rB  r   niter_sado_cgr   r@  freefull_outputr>  rF  r   r   methr  rY   packedr   rsltnext_methodr  r?  gnr5  vcomp_unscaledrT   rV  r/  rV   rW   f1f2rU  	hess_diagrW  patr   rX  params_packedrY  r   s)                                           r   r-  zMixedLM.fit"  s   d3 3 3"" 	I 	IA''CaGHHH>,,,FF$$ 	XF 	B 	BDzz||000 9D@B B B 1 	 	DDD"49diCCF!x	22FF49--FM749--FLL,66 =% |$$	DJ(>(JJJ*66$diDM# 7 % %FF &&$*ty*@@@*66$diDM$ 7 & &FF %%;<<< 	.#'t#3Jx Z''%*
6"&&e&LLF!||=>>> 3v;;'' ; ;"uww{ 104*0)1 1 &01 1 #K0 Eq53v;;&&"(Q-KM?+M*, , , ,DCM#'9:::: ]4;//FD,---$[1	 	3DK((BA''B=BCM#1222
 **DIty4= + P P -,,_nMM	4$y/>JJ(&i!mI"&)@)@"A"AD"Hi!m?"&"7"7$"> 	3 	3JCM#1222 \\&))
d 	*M.)))GDMM	=&&D-**E$*GGC$$B!"I2ww{{RVB^^,')y}}eV'<'<RVB^^$9==$''D6)q.!! 	3/CM#1222 ))5)FF }dUlCC &%"1#'96$%y,yy
y=-$W---r   c                 &    t          | |||          S r'   )_mixedlm_distribution)r+   rY   r/  r   s       r   get_distributionzMixedLM.get_distribution  s    $T65$???r   )NNTr   )NNNFr   r'   )Nr#  r   r$  r%  r&  )r[  )T)	NTr   TNNNFN)r-   r.   r/   r0   r,   r   classmethodr  r  r   rZ  rm  r   r2  r   r  r  r  r  r  r6  r4  r-  r  __classcell__r   s   @r   r   r   a  s       U Un 596<H0 H0 H0 H0 H0 H0TO O O0 EI<BH H H H H [HT$ $ $ $>0 0 0  EF47^. ^. ^. ^.@N N N N`: : :x  6  4k k k kZ' ' ' 'R'B 'B 'B 'BRZ, Z, Z,x+, +, +, +,Zn n n`2 2 2h :;8<&*J. J. J. J. J. J.X@ @ @ @ @ @ @r   r   c                       e Zd ZdZd Zd ZdS )r  aW  
    A private class for simulating data from a given mixed linear model.

    Parameters
    ----------
    model : MixedLM instance
        A mixed linear model
    params : array_like
        A parameter vector defining a mixed linear model.  See
        notes for more information.
    scale : scalar
        The unexplained variance
    exog : array_like
        An array of fixed effect covariates.  If None, model.exog
        is used.

    Notes
    -----
    The params array is a vector containing fixed effects parameters,
    random effects parameters, and variance component parameters, in
    that order.  The lower triangle of the random effects covariance
    matrix is stored.  The random effects and variance components
    parameters are divided by the scale parameter.

    This class is used in Mediation, and possibly elsewhere.
    c                    || _         ||n|j        | _        t                              ||j        |j        dd          }|j        | _        ||j        z  | _        ||j        z  | _        || _	        t          j        |j        t                    }t          |j                  D ]\  }}|||j        |         <   || _        d S )NFTr   )modelr   rG   r^   rJ   r:   rT   rV   rW   r/  r   rS   r   rQ   r   r   r   	group_idx)	r+   r  rY   r/  r   por  r8   r   s	            r   r,   z_mixedlm_distribution.__init__	  s    
 ,DD%*	&&
EJt= = bi'RX%

HUZs333	e011 	0 	0DAq./Ie'*++"r   c           
         | j         }t          j        | j        | j                  }t          j                            |j        |j        f          }t          j        |t          j	        
                    | j                  j                  }||| j        ddf         |j        z                      d          z  }t!          |j        j                  D ]\  }}|j        j        |         }| j        |         }t!          |j                  D ]}\  }	}
||	         }|j        |
         }t          j                            |j        d                   }||xx         t          j        |          t          j        ||          z  z  cc<   ~|t          j        | j                  t          j                            t5          |                    z  z  }|S )z
        Return a vector of simulated values from a mixed linear
        model.

        The parameter n is ignored, but required by the interface
        )re  Nr   )r  r   r   r   rT   randomnormalr   r:   rh   ri   rV   r   r  rC   r!   r   r   r(   r*   rW   r   r   r   rk   r/  rR   )r+   nr  r   rR  r   r   r~  r   r   r   exgr   s                r   rvsz_mixedlm_distribution.rvs 	  s    
 F49dn-- I5>5:">??F1bi((55788	a!"U]277::: em122 	5 	5DAq#A&B
1A!%"455 5 51e&q)I$$#)A,$77"bfS!nn44	5 	
RWTZ  29#3#3Q#3#@#@@@r   N)r-   r.   r/   r0   r,   r  r1   r   r   r  r    s<         6# # #$    r   r  c                       e Zd ZdZ fdZed             Zed             Zed             Zed             Z	d Z
ed             Zed	             Zd fd	Z	 	 ddZed             Zed             Zed             Z	 	 ddZ xZS )r<  a  
    Class to contain results of fitting a linear mixed effects model.

    MixedLMResults inherits from statsmodels.LikelihoodModelResults

    Parameters
    ----------
    See statsmodels.LikelihoodModelResults

    Attributes
    ----------
    model : class instance
        Pointer to MixedLM model instance that called fit.
    normalized_cov_params : ndarray
        The sampling covariance matrix of the estimates
    params : ndarray
        A packed parameter vector for the profile parameterization.
        The first `k_fe` elements are the estimated fixed effects
        coefficients.  The remaining elements are the estimated
        variance parameters.  The variance parameters are all divided
        by `scale` and are not the variance parameters shown
        in the summary.
    fe_params : ndarray
        The fitted fixed-effects coefficients
    cov_re : ndarray
        The fitted random-effects covariance matrix
    bse_fe : ndarray
        The standard errors of the fitted fixed effects coefficients
    bse_re : ndarray
        The standard errors of the fitted random effects covariance
        matrix and variance components.  The first `k_re * (k_re + 1)`
        parameters are the standard errors for the lower triangle of
        `cov_re`, the remaining elements are the standard errors for
        the variance components.

    See Also
    --------
    statsmodels.LikelihoodModelResults
    c                     t                                          |||           | j        j        | _        | j        t          j                            | j        j                  z
  | _        d S )N)normalized_cov_params)	r   r,   r  r   r   rh   matrix_rankr   df_resid)r+   r  rY   
cov_paramsr   s       r   r,   zMixedLMResults.__init__k	  sS    jIIIJO		BI$9$9$*/$J$JJr   c           	      0   t          j        | j        j        | j                  }| j        }t          | j        j                  D ]\  }}| j        j        |         }g }| j        j	        %|
                    | j        j	        |                    t          | j                  D ]2}|
                    | j        j        j        |         |                    3t          j        |d          }||xx         t          j        |||                   z  cc<   |S )z
        Returns the fitted values for the model.

        The fitted values reflect the mean structure specified by the
        fixed effects and the predicted random effects.
        Nr   r^  )r   r   r  r   rT   random_effectsr   r   r   r   r   rB   rL   r   r*   rl   )r+   r-  r  r  r  r]   r  r   s           r   fittedvalueszMixedLMResults.fittedvaluesq	  s    fTZ_dn55 ()@AA 
	. 
	.OHe'.BCz$0

4:0:;;;49%% A A

4:-215h?@@@@.1---CGGGrvc2e9---GGGG
r   c                 *    | j         j        | j        z
  S )z
        Returns the residuals for the model.

        The residuals reflect the mean structure specified by the
        fixed effects and the predicted random effects.
        )r  r   r  r+   s    r   rQ  zMixedLMResults.resid	  s     z$"333r   c                     | j         j        j        d         }t          j        t          j        |                                           d|                   S )zb
        Returns the standard errors of the fixed effect regression
        coefficients.
        r   r   )r  r   r   r   rk   rU   r  r+   r   s     r   bse_fezMixedLMResults.bse_fe	  sB     JO!!$wrwt0011!A#6777r   c                     | j         j        j        d         }t          j        | j        t          j        |                                           |d         z            S )a)  
        Returns the standard errors of the variance parameters.

        The first `k_re x (k_re + 1)` elements of the returned array
        are the standard errors of the lower triangle of `cov_re`.
        The remaining elements are the standard errors of the variance
        components.

        Note that the sampling distribution of variance parameters is
        strongly skewed unless the sample size is large, so these
        standard errors may not give meaningful confidence intervals
        or p-values if used in the usual way.
        r   N)r  r   r   r   rk   r/  rU   r  r#  s     r   bse_rezMixedLMResults.bse_re	  sI     JO!!$wtzBGDOO,=,=$>$>qrr$BBCCCr   c                    t          | j        j        j                  }t	          | j        j        j                  D ]E\  }| j        j        j        |         |         }fd|D             }|                    |           F|S )Nc                      g | ]
} d | dS )[]r1   )r7   ry   r   s     r   r9   z3MixedLMResults._expand_re_names.<locals>.<listcomp>	  s%    ***!Q+++++***r   )	rA   r  r   r   r   r   r(   r)   r   )r+   r  r(   r   vgnar   s         @r   _expand_re_nameszMixedLMResults._expand_re_names	  s    TZ_233dj0677 	 	DAq#,Q/9B****r***BLLr   c           	      `   	 t           j                            | j                  }n'# t           j        j        $ r t          d          w xY w| j        }| j        }i }t          | j	        j
                  D ]5\  }}| j	        j        |         }| j	        j        |         }| j	        j        |         }	| j	        j        |         }
| j	                            ||          }|}| j        dk    rt          j        || j                  }||z
  }t'          | j        |	|
|d|z            } ||          }t+          |	j        |          }t          j        | j        |d|                   |d|<   ||dxx         |z  cc<   t/          j        ||                     |                    ||<   7|S )a*  
        The conditional means of random effects given the data.

        Returns
        -------
        random_effects : dict
            A dictionary mapping the distinct `group` values to the
            conditional means of the random effects for the group
            given the data.
        zACannot predict random effects from singular covariance structure.r   r   N)rr  )r   rh   r0  rV   rj   r   rW   r:   r   r  r   r   r   r   r   r2  rJ   r   rT   r   r/  r   r   r;   r?   r-  )r+   rH  rW   r:   
ranef_dictr  r  r   r   rO  rP  rN  rQ  rK  rz   r  r  s                    r   r  zMixedLMResults.random_effects	  s   	?t{33JJy$ 	? 	? 	? > ? ? ?	? 
y
()@AA 	> 	>OHeJ'1E:%h/D:$X.DJ&x0EZ--eX>>F Ey1}}dn55 T5*!"V- -F&--C%%EF4;af>>E!D&M$%%LLLF"LLL "	T228<<!> !> !>Ju s	   $' $Ac           	      \   	 t           j                            | j                  }n# t           j        j        $ r d}Y nw xY w| j        }i }t          | j        j                  D ]}| j        j	        |         }| j        j
        |         }| j        j        |         }| j                            ||          }t          | j        |||d|z            }	|j        d         }
| j        j        d         }t          j        |
|t#          |          z   f          }t          j        |ddd|f         | j                  |ddd|f<   t          j        |dd|df         t          j        |                    |dd|df<    |	|          }t          j        |j        |          }| }|d|d|fxx         | j        z  cc<   t          j        ||j        d                   }|||fxx         |z  cc<   |                     |          }t/          j        |||          }|||<   |S )ak  
        Returns the conditional covariance matrix of the random
        effects for each group given the data.

        Returns
        -------
        random_effects_cov : dict
            A dictionary mapping the distinct values of the `group`
            variable to the conditional covariance matrix of the
            random effects given the data.
        Nr   r   )rr  r=   )r   rh   r0  rV   rj   rW   rB   r  r   r   r   r   r2  r   r/  r   ra   rR   r   rU   r   aranger-  r;   r<   )r+   rH  rW   r/  r  rO  rP  labelrN  rz   r  r   mat1mat2r   r]   r,  s                    r   random_effects_covz!MixedLMResults.random_effects_cov	  s3   	t{33JJy$ 	 	 	JJJ	 

dj122 	" 	"H:$X.DJ&x0EJ+H5EZ--eX>>F T5*!"V- -F 
1A!!$A8QCKK011D6$qqq!A#v,<<DAaCL&aaaebgfoo>>DABBK6$<<D6$&$''DAac1Q3hKKK4;&KKK1agaj))Bb"fIIIIII&&x00BQb"555A !Jus   $' A A Nc                 8   |j         d         | j        k    rt          d| j        z            | j        | j        z   }t          j        |j         d         |f          }t          j        ||fd          }t                      	                    ||          }|S )a  
        Compute a t-test for a each linear hypothesis of the form Rb = q

        Parameters
        ----------
        r_matrix : array_like
            If an array is given, a p x k 2d array or length k 1d
            array specifying the linear restrictions. It is assumed
            that the linear combination is equal to zero.
        scale : float, optional
            An optional `scale` to use.  Default is the scale specified
            by the model fit.
        use_t : bool, optional
            If use_t is None, then the default of the model is used.
            If use_t is True, then the p-values are based on the t
            distribution.
            If use_t is False, then the p-values are based on the normal
            distribution.

        Returns
        -------
        res : ContrastResults instance
            The results for the test are attributes of this results instance.
            The available results have the same elements as the parameter table
            in `summary()`.
        r   z*r_matrix for t-test should have %d columnsr   r^  )use_t)
r   rJ   r   rK   rL   r   rS   rl   r   t_test)r+   r_matrixr7  r   z0tst_rsltr   s         r   r8  zMixedLMResults.t_test
  s    6 >!	))I#y) * * * J"Xx~a(!,-->8R.q99977>>(%>88r   皙?c           
      	   ddl m} |                                }i }d|d<   || j        j        }| j        j        j        dd         }	t          | j                  }
t          |	          t          | j                  z
  }|.t          |          |
k    rd|
z  }t          |          ||	d|
<   |.t          |          |k    rd|z  }t          |          ||	|
d<   t          | j        j                  |d<   t          | j        j                  |d	<   t          j        d
 | j        j        D                       }dt!          |          z  |d<   dt#          |          z  |d<   dt          j        |          z  |d<   ||d<   | j        |d<   | j        |d<   | j        |d<   | j        rdnd|d<   |                    |           |                    d           dt          j        t          j        | j        | j        z   | j        z   df          z  }| j        |d| j        df<   t          j        t          j        |                                  d| j                                     |d| j        df<   |d| j        df         |d| j        df         z  |d| j        df<   dtC          j"        t          j#        |d| j        df                              z  |d| j        df<   tC          j$        |dz             }|d| j        df         ||d| j        df         z  z
  |d| j        df<   |d| j        df         ||d| j        df         z  z   |d| j        df<   | j        }tK          | j&                  D ]\}tK          |dz             D ]G}| j'        ||f         ||df<   t          j        | j                  | j(        |         z  ||df<   |dz  }H]tK          | j                  D ]E}| j)        |         ||df<   t          j        | j                  | j(        |         z  ||df<   |dz  }FtU          j+        |	|          }d d!d"d#d$t          |dz            z   t          d|dz  z
            d%z   g|_,        |j,        D ]}fd&||         D             ||<   |-                    |d'(           |S ))aA  
        Summarize the mixed model regression results.

        Parameters
        ----------
        yname : str, optional
            Default is `y`
        xname_fe : list[str], optional
            Fixed effects covariate names
        xname_re : list[str], optional
            Random effects covariate names
        title : str, optional
            Title for the top table. If not None, then this replaces
            the default title
        alpha : float
            significance level for the confidence intervals

        Returns
        -------
        smry : Summary instance
            this holds the summary tables and text, which can be
            printed or converted to various output formats.

        See Also
        --------
        statsmodels.iolib.summary2.Summary : class to hold summary results
        r   )summary2r   zModel:Nz&xname_fe should be a list of length %dz&xname_re should be a list of length %dzNo. Observations:zNo. Groups:c                 ,    g | ]}t          |          S r1   )rR   r   s     r   r9   z*MixedLMResults.summary.<locals>.<listcomp>{
  s    ;;;!s1vv;;;r   z%.0fzMin. group size:zMax. group size:z%.1fzMean group size:zDependent Variable:zMethod:zScale:zLog-Likelihood:YesNoz
Converged:z%Mixed Linear Model Regression Resultsz%.3f   r   rI            )rr  r   zCoef.zStd.Err.zzP>|z|r)  r*  c                 F    g | ]}t          j        |          r|z  nd S ) )r   isfinite)r7   r   	float_fmts     r   r9   z*MixedLMResults.summary.<locals>.<listcomp>
  s>     + + + *,Q?	AR + + +r   r)align).statsmodels.iolibr>  Summaryr  endog_namesr   r   rR   rT   r   r   r   r   r   rX   r   rb  r3  meanr>  r/  llfr?  add_dict	add_titler8  r   rJ   rK   rL   rk   rU   r  r   cdfr1  ppfrB   r:   rV   bserW   r;   r<   r=   add_df)r+   ynamexname_fexname_retitler+  r>  smryinfor   k_fe_paramsk_re_paramsr  gssdfqmr   r   r   colrJ  s                       @r   summaryzMixedLMResults.summaryB
  s   < 	/.....!!"X=J*Ejo1!!!4$.))+&&T^)<)<<8}}++>L oo%(0K%8}}++>L oo%(0K%$'
(;$<$< !!$*"566]X;;tz':;;;<<#)CGG#3 #)CGG#3 #)BGBKK#7 &+"#+YX"&(&*n>UU$\d>???	frw	DJ 6 BAFGGG #nAdiKN !gbgdoo.?.?$).L&M&MNNAdiKN "!DI+q.1C$)Q4GGAdiKN  $(BF3q{A~3F,G,G+G"H"HHAdiKN huqy!!!!!DI+q.1BQty[!^9L4LLAdiKN!!DI+q.1BQty[!^9L4LLAdiKN Yty!! 	 	A1q5\\  ![A.BE
WTZ0048B<?BE
a ty!! 	 	AACAJ,,tx|;CAJ!GBBl3777
CSq\\)3qqy>>C+?A; 	+ 	+C+ + + +!$S+ + +CHH 	Cs###r   c                 D    | j                             | j        d          S )NF)r  )r  r  r=  r!  s    r   rQ  zMixedLMResults.llf
  s     z!!$"4!GGGr   c                     | j         rt          j        S | j        2| j                            dd                                          dz   }n| j        j        dz   }d| j        |z
  z  S )zAkaike information criterionNFTr  r   )	r   r   r8  r  ro   r!   rY   re  rQ  r+   dfs     r   aiczMixedLMResults.aic
  sj     9 	6M<#((%(EEIIKKaOBB!A%BTX]##r   c                    | j         rt          j        S | j        2| j                            dd                                          dz   }n| j        j        dz   }d| j        z  t          j	        | j
                  |z  z   S )zBayesian information criterionNFTr  r   rg  )r   r   r8  r  ro   r!   rY   re  rQ  r   r   rh  s     r   biczMixedLMResults.bic
  s{     9 	6M<#((%(EEIIKKaOBB!A%BDH}rvdi002555r   rE  r`  c           	         | j         }|j        }	|j        }
|j        }|j        |j        }}|dk    rt          j        |
          }||d<   d||<   |j        	                                dd|f         }| j
        	                                }|j        }|t          j        ||                   }||_        |d         }| j        |z  }|d         |z
  | j        z  }|d         |z   | j        z  }n|dk    rz| j         j        j                            |          }| j
        	                                }| j        }||         |z
  | j        z  }||         |z   | j        z  }||         | j        z  }|dk    rt%          d          t          j        |||dz             }t          j        |||dz             dd         }t          j        ||f          }t+          |	|
|          }| j        Dt          j        |	          |_        t          j        |          }t          j        |
|
f          }nJ| j        j        |_        | j        j        }| j        j        }|dk    r|t          j        ||                   }|dk    rd|d<   nd||<   ||_        ||_        | j         j        }|                                }|dk    r||d<   g }|D ]} |||fi |} |dk    r&|j        	                                }||d<   ||_        n
||j        |<    | j        d
||| j        | j        d	|j        }!|                    ||!j        z  |!j         g           t          j!        |          }|S )a	  
        Profile-likelihood inference for variance parameters.

        Parameters
        ----------
        re_ix : int
            If vtype is `re`, this value is the index of the variance
            parameter for which to construct a profile likelihood.  If
            `vtype` is 'vc' then `re_ix` is the name of the variance
            parameter to be profiled.
        vtype : str
            Either 're' or 'vc', depending on whether the profile
            analysis is for a random effect or a variance component.
        num_low : int
            The number of points at which to calculate the likelihood
            below the MLE of the parameter of interest.
        dist_low : float
            The distance below the MLE of the parameter of interest to
            begin calculating points on the profile likelihood.
        num_high : int
            The number of points at which to calculate the likelihood
            above the MLE of the parameter of interest.
        dist_high : float
            The distance above the MLE of the parameter of interest to
            begin calculating points on the profile likelihood.
        **fit_kwargs
            Additional keyword arguments passed to fit.

        Returns
        -------
        An array with two columns.  The first column contains the
        values to which the parameter of interest is constrained.  The
        second column contains the corresponding likelihood values.

        Notes
        -----
        Only variance parameters can be profiled.
        r  r   Nr`   vczSdist_low is too large and would result in a negative variance. Try a smaller value.r   rC   )rB  r  r   r@  r1   )"r  rJ   r:   rL   r   r   r   r1  rC   re   r=  rV   r;  r/  r   r(   rr  rW   r   linspacerl   rG   r  r   rT   r   _get_init_kwdsr-  r   r@  _resultsr   rQ  r   )"r+   re_ixvtypenum_lowdist_lownum_high	dist_highrF  pmodelrJ   r:   rL   r   r   r]   rC   rY   r5  ru0rV   lowhighrW   leftrightrvaluesr  r  klassinit_kwargslikevr   r  r  s"                                     r   
profile_rezMixedLMResults.profile_re
  s   R {{{lFKt D==4BBqEBuIn))++AAArE2G ',,..F$mO-bfRnn=O+FM!$'C Z/1F$<(*dj8C4L9,
:DDd]]J&,22599E',,..FJE<(*dj8C%L9,
:D,+C !88 G H H H{3Wq[11Cxz221226.$// T4..<WT]]DNGDMME'4,''CC "\3DNL&E,%C}}"&R..)D==CIIE%L

$++--D==%,K	" 	5 	5AE%5555E}}++-- t &&'U# 59 +&t"&)T\+ +)+ ++3  LL!dj.$(34444
5!!r   r'   )NNNNr<  )rE  r`  rE  r`  )r-   r.   r/   r0   r,   r   r  rQ  r$  r&  r-  r  r5  r8  rd  rQ  rj  rl  r  r
  r  s   @r   r<  r<  B	  s       & &PK K K K K   ^. 4 4 ^4 8 8 ^8 D D ^D"   . . ^.` / / ^/f# # # # # #J ;?"%r r r rh H H ^H $ $ ^$ 6 6 ^6 IJD D D D D D D Dr   r<  c                       e Zd ZddddddZej        j        Zej        	                    ee          Zi Z
ej        j        Zej        	                    e
e          ZdS )rA  )generic_columnsr   )r  xnames)generic_columns_2dr   )r&  rT   r$  rV   r5  N)r-   r.   r/   _attrsbaseLikelihoodResultsWrapper_wrap_attrs_upstream_attrswrapunion_dicts_methods_wrap_methods_upstream_methodsr1   r   r   rA  rA  W  sv        A85?!H	 F 3?O)''@@KH5CI))(4EFFMMMr   rA  c                    t                      }|g}||                    |           |'|                    |                                           ddlm ddlm} dd l}h d}	|D ]K}
 ||
          fd}|	                    |          }|D ] }||	vr|
                    |j                   !Lt          |t          | j                  z            }| |         } t          j        |                               d          }t#          |          t$          ur|t          j        |          z  }| j        |d d f         |t)          j        |                   fS )Nr   )	asunicode)StringIO>   ***()+-/:c                  D                                     }  | d          S )Nascii)readline)liner  rls    r   rluz_handle_missing.<locals>.rluz  s"    ;;==D9T7+++r   r   )r   r   r   valuesstatsmodels.compat.pythonr  ior  tokenizegenerate_tokensaddstringr  r=   r;   notnullallr  r   r	  r   r   )r   r   r   r  r  tokensformsr  r  skiptoksfmlr  r   tokr   r  r  s                  @@r   r   r   f  s   UUFIEZ   Z&&(()))333333OOO888H 
' 
'Xc]]	, 	, 	, 	, 	, 	, $$S)) 	' 	'C(""

3:&&&	' FS...//F<D	D				a	 	 BF||3
bj   8BE?F2:b>>222r   )*r0   r   numpyr   pandasr;   r  scipyr   scipy.stats.distributionsr   statsmodels.base._penaltiesr   statsmodels.base.modelr  r  statsmodels.toolsr   r   statsmodels.tools.decoratorsr   statsmodels.tools.sm_exceptionsr   r7  r   r   r#   r%   rE   rG   r   r   r   LikelihoodModelr   r  LikelihoodModelResultsResultMixinr<  r  rA  r   r1   r   r   <module>r     s  O O`                 * * * * * * / / / / / / % % % % % % % % % 0 0 0 0 0 0 7 7 7 7 7 7 > > > > > >D
 
 
# # #(
, 
, 
,       .  &{ { { { { { { {|E E EP< < <~2 2 2BN@ N@ N@ N@ N@d" N@ N@ N@b4M M M M M M M M`R R R R RT0$2B R R RjG G G G GD9 G G G"3 "3 "3 "3 "3r   