M/PhIdZddlZddlZddlZddlmZddZdZ dZ dZ d Z d Z d Zd Zd ZddddejejdfdZdS)aS Implementation of Regression on Order Statistics for imputing left- censored (non-detect data) Method described in *Nondetects and Data Analysis* by Dennis R. Helsel (John Wiley, 2005) to estimate the left-censored (non-detect) values of a dataset. Author: Paul M. Hobson Company: Geosyntec Consultants (Portland, OR) Date: 2016-06-14 N)statsFc||||d}||||d}||||krB|||||k}|rd}tj|t j||gd}|||gdS)a This function prepares a dataframe for ROS. It sorts ascending with left-censored observations first. Censored observations larger than the maximum uncensored observations are removed from the dataframe. Parameters ---------- df : DataFrame observations : str Name of the column in the dataframe that contains observed values. Censored values should be set to the detection (upper) limit. censorship : str Name of the column in the dataframe that indicates that a observation is left-censored. (i.e., True -> censored, False -> uncensored) Returns ------ sorted_df : DataFrame The sorted dataframe with all columns dropped except the observation and censorship columns. raxiszKDropping censored observations greater than the max uncensored observation.T)drop) sort_valuesmaxwarningswarnpdconcat reset_index)df observations censorshipr censored uncensoredmsgcombineds Z/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/imputation/ros.py _ros_sortrs<"Z.!--l-CCHR ^O$00A0FFJ !!##j&>&B&B&D&DDDH\2j6N6R6R6T6TTU  5C M#   y(J/a888H \:. / ; ; ; F FFcfd}fd}fd}d}d}}tjj|f} | | jddkrJ| kr.t j| g} tj| dg} || | jd d d f<| |d | jd d d f<| |d | jd d df<| |d | jd d df<| t| jdd z} || d | d| jd d df<n;gd} tjt j dt| f| } | S)aM Computes the Cohn numbers for the detection limits in the dataset. The Cohn Numbers are: - :math:`A_j =` the number of uncensored obs above the jth threshold. - :math:`B_j =` the number of observations (cen & uncen) below the jth threshold. - :math:`C_j =` the number of censored observations at the jth threshold. - :math:`\mathrm{PE}_j =` the probability of exceeding the jth threshold - :math:`\mathrm{DL}_j =` the unique, sorted detection limits - :math:`\mathrm{DL}_{j+1} = \mathrm{DL}_j` shifted down a single index (row) Parameters ---------- dataframe : DataFrame observations : str Name of the column in the dataframe that contains observed values. Censored values should be set to the detection (upper) limit. censorship : str Name of the column in the dataframe that indicates that a observation is left-censored. (i.e., True -> censored, False -> uncensored) Returns ------- cohn : DataFrame c|dk}|dk}}||z|zjdS)zD A, the number of uncensored obs above the given threshold. lower_dlupper_dlrshape)rowabovebelowdetectrrrs r nuncen_abovez"cohn_numbers..nuncen_abovehsX < C O3< 3z?2Z.%%-&()/22rc |dk} |dk}}}||zjd}||zjd}||zS)zW B, the number of observations (cen & uncen) below the given threshold rrr) r less_thanless_thanequalrr LTE_censored LT_uncensoredrrrs r nobs_belowz cohn_numbers..nobs_belowxs |$s:6 L)S_<n_ j>.834:1= 9z128; m++rcp}|}||dk}|S)zP C, the number of censored observations at the given threshold. r)sum)rcensored_index censored_datacensored_belowrrrs r ncen_equalz cohn_numbers..ncen_equals= J<(8 &#j/9!!###rc|jddkr9|ddtjStjgS)z: Sets the upper_dl DL for each row of the Cohn dataframe. rr)value)rshiftfillnanpinf)cohns rset_upper_limitz%cohn_numbers..set_upper_limitsH :a=1   #))"--44264BB BF8Orct|}tj|d}d|d<t|dz ddD]:}||dzd||dzz ||z||||zz z||<;|S)zS Computes the probability of excedance for each row of the Cohn dataframe. float64)dtypegr2r1)lenr6emptyrange)ABNPEjs r compute_PEz cohn_numbers..compute_PEs FF Xay ) ) )2qsB## C CAqsGq2ac7{ad2adQqTkBBBqEE rrr)columnsNrr1rr#r)r/prob_exceedance)rrr#r)r/rH)r uniquelocsortrminr6hstack DataFrameapplyreindexr@r?r>) rrrr#r)r/r9rFr-DLsr8dl_colss ``` r cohn_numbersrSCsTJ3333333 ,,,,,,,0$$$$$$$   zNM )BF=,67 8 8CHHJJJ y|a l    ! !CGGII - -)R -1133S9::C |C*666"1/$"7"7J&*jjAj&F&FN"#$(JJzJ$B$BL!$(JJzJ$B$BL!||E#)A,"23344)3D4H$|J\)])]%%&&BBB|BHaW%677III Krc~|jddkr)tj|d|k\}|d}nd}|S)a Locates the corresponding detection limit for each observation. Basically, creates an array of indices for the detection limits (Cohn numbers) corresponding to each data point. Parameters ---------- obs : float A single observation from the larger dataset. cohn : DataFrame DataFrame of Cohn numbers. Returns ------- det_limit_index : int The index of the corresponding detection limit in `cohn` See Also -------- cohn_numbers rrr2)rr6where)obsr8indexdet_limit_indexs r_detection_limit_indexrYsF2 z!}q$z*c122) rc|}d|jdddf<|||gdd}|S)a Ranks each observation within the data groups. In this case, the groups are defined by the record's detection limit index and censorship status. Parameters ---------- df : DataFrame dl_idx : str Name of the column in the dataframe the index of the observations' corresponding detection limit in the `cohn` dataframe. censorship : str Name of the column in the dataframe that indicates that a observation is left-censored. (i.e., True -> censored, False -> uncensored) Returns ------- ranks : ndarray Array of ranks for the dataset. r1Nrank)byc*|SN)cumsum)gs rz!_ros_group_rank.. s!((**r)copyrJgroupby transform)rdl_idxrrankss r_ros_group_rankrgs]: GGIIEEIaaai &*- ..v6i,,--  Lrc|d}|d}||}|j|}|j|dz}|rd|dz |z|ddzz Sd|dz |d|dz |z|ddzz zS)a ROS-specific plotting positions. Computes the plotting position for an observation based on its rank, censorship status, and detection limit index. Parameters ---------- row : {Series, dict} Full observation (row) from a censored dataset. Requires a 'rank', 'detection_limit', and `censorship` column. censorship : str Name of the column in the dataframe that indicates that a observation is left-censored. (i.e., True -> censored, False -> uncensored) cohn : DataFrame DataFrame of Cohn numbers. Returns ------- plotting_position : float See Also -------- cohn_numbers rXr[r1rHr/r#)iloc)rrr8DL_indexr[rdl_1dl_2s r _ros_plot_posrms<$%H v;D:H 9X D 9X\ "D0D*++t3tL7I!7KLLD*++5F0G$O`Ja0a0^,Q.000 0rcrtj|d\}}tj|S)z Computes standard normal (Gaussian) plotting positions using scipy. Parameters ---------- observations : array_like Sequence of observed quantities. Returns ------- plotting_position : array of floats F)fit)rprobplotnormcdf)rppos sorted_ress r_norm_plot_posru=s1~l>>>D* :>>$  rc|fdd}||}tj|d}|||j|j|<|S)aP Compute the plotting positions for the observations. The ROS-specific plotting postions are based on the observations' rank, censorship status, and corresponding detection limit. Parameters ---------- df : DataFrame censorship : str Name of the column in the dataframe that indicates that a observation is left-censored. (i.e., True -> censored, False -> uncensored) cohn : DataFrame DataFrame of Cohn numbers. Returns ------- plotting_position : array of float See Also -------- cohn_numbers c&t|Sr^)rm)rrr8s rraz$plotting_positions..js-:t"D"Drr1rW) requirements)rOr6requirerKrJrW)rrr8plot_pos ND_plotposND_plotpos_arrs `` rplotting_positionsrNs8xxDDDDD1xMMH"Z.)JZ ===N9GHLJ%bn56 Orch||}||}tj|d|||||}|dd\}} |||d|z| z|jdddf<tj|||d|||jdddf<|S)a Executes the basic regression on order stat (ROS) proceedure. Uses ROS to impute censored from the best-fit line of a probability plot of the uncensored values. Parameters ---------- df : DataFrame observations : str Name of the column in the dataframe that contains observed values. Censored values should be set to the detection (upper) limit. censorship : str Name of the column in the dataframe that indicates that a observation is left-censored. (i.e., True -> censored, False -> uncensored) transform_in, transform_out : callable Transformations to be applied to the data prior to fitting the line and after estimated values from that line. Typically, `np.log` and `np.exp` are used, respectively. Returns ------- estimated : DataFrame A new dataframe with two new columns: "estimated" and "final". The "estimated" column contains of the values inferred from the best-fit line. The "final" column contains the estimated values only where the original observations were censored, and the original observations everwhere else. ZprelimNr= estimatedfinal)r linregressrJr6rU) rrr transform_in transform_outuncensored_mask censored_mask fit_paramsslope intercepts r_imputerusD*~oOzNM! 9 o& R %o677J ""1"~E9 +]52i=3O+OR[+[\\BF111k>"Z."[/2lCSTTBF111g: Irct|||}t|||}||t|f|jdddf<t |d||jdddf<t ||||jdddf<tj |d|jdddf<t|||||S)a DataFrame-centric function to impute censored valies with ROS. Prepares a dataframe for, and then esimates the values of a censored dataset using Regression on Order Statistics Parameters ---------- df : DataFrame observations : str Name of the column in the dataframe that contains observed values. Censored values should be set to the detection (upper) limit. censorship : str Name of the column in the dataframe that indicates that a observation is left-censored. (i.e., True -> censored, False -> uncensored) transform_in, transform_out : callable Transformations to be applied to the data prior to fitting the line and after estimated values from that line. Typically, `np.log` and `np.exp` are used, respectively. Returns ------- estimated : DataFrame A new dataframe with two new columns: "estimated" and "final". The "estimated" column contains of the values inferred from the best-fit line. The "final" column contains the estimated values only where the original observations were censored, and the original observations everwhere else. )rr)argsNrXr[r|r) rSrrOrYrJrgrrrqppfr)rrrrrr8modeleds r_do_rosrsJ * M M MD*MMMG(/ (=(C(CDZbfah(C(i(iGK$$%,W6GTTGK6 !3GZ!N!NGK:  % wz/B C CGK9  7L*lM R RRrr=g?g?Tc 8|tj||d}d}d}|jd} ||t} | | z } | | z } | dkr1|||g} ||| jdddf<nm| |ks| |krN|||g} ||| jdddf<| j||dfxx|zcc<nt|||||} |r | dj } | S)a) Impute censored dataset using Regression on Order Statistics (ROS). Method described in *Nondetects and Data Analysis* by Dennis R. Helsel (John Wiley, 2005) to estimate the left-censored (non-detect) values of a dataset. When there is insufficient non-censorded data, simple substitution is used. Parameters ---------- observations : str or array-like Label of the column or the float array of censored observations censorship : str Label of the column or the bool array of the censorship status of the observations. * True if censored, * False if uncensored df : DataFrame, optional If `observations` and `censorship` are labels, this is the DataFrame that contains those columns. min_uncensored : int (default is 2) The minimum number of uncensored values required before ROS can be used to impute the censored observations. When this criterion is not met, simple substituion is used instead. max_fraction_censored : float (default is 0.8) The maximum fraction of censored data below which ROS can be used to impute the censored observations. When this fraction is exceeded, simple substituion is used instead. substitution_fraction : float (default is 0.5) The fraction of the detection limit to be used during simple substitution of the censored values. transform_in : callable (default is np.log) Transformation to be applied to the values prior to fitting a line to the plotting positions vs. uncensored values. transform_out : callable (default is np.exp) Transformation to be applied to the imputed censored values estimated from the previously computed best-fit line. as_array : bool (default is True) When True, a numpy array of the imputed observations is returned. Otherwise, a modified copy of the original dataframe with all of the intermediate calculations is returned. Returns ------- imputed : {ndarray, DataFrame} The final observations where the censored values have either been imputed through ROS or substituted as a fraction of the detection limit. Notes ----- This function requires pandas 0.14 or more recent. N)rVcenrVrrr) r rNrastypeintr+rbrJrvalues)rrrmin_uncensoredmax_fraction_censoredsubstitution_fractionrras_arrayN_observations N_censored N_uncensoredfraction_censoredoutputs r impute_rosrs\H z \,zBB C C  Xa[NJ&&s++//11J!J.L"^3 Q\:./4466!#L!1 111g:  ' '->AV-V-V\:./4466!#L!1 111g: 2j>7*+++/DD++++ \:|]SS(' Mr)F)__doc__r numpyr6pandasr scipyrrrSrYrgrmrurrrlogexprrrrs  *G*G*G*GZDDDND###L(0(0(0V   "$$$N444n/S/S/Sd-1%(F"&hhhhhhr