0Ph2ddlZddlZddlmZddlmZddlmZddlZ ddl m Z ddl m ZddlmZmZmZddlmZdd lmZmZdd lmZmZdd lmZdd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$dZ%dZ&GddeeZ'Gdde'Z(GddeeZ)dS)N)Counter)partial)Callable)sparse) BaseEstimatorTransformerMixin _fit_context) _get_mask) is_pandas_na is_scalar_nan) MissingValues StrOptions)_mode) _get_median) FLOAT_DTYPES_check_feature_names_in_check_n_featurescheck_is_fitted validate_datact|rdS|jjdvrOt|tjs7t d|jt|dSdS)N)fiuzn'X' and 'missing_values' types are expected to be both numerical. Got X.dtype={} and type(missing_values)={}.) r dtypekind isinstancenumbersReal ValueErrorformattype)Xmissing_valuess T/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/sklearn/impute/_base.py_check_inputs_dtyper&suN##w|&&z.',/W/W& ((.qw^8L8L(M(M   '&&&c|jdkr|jtkr^t|}|dddt fd|D}n0t|}|dd}|ddnd}ddkr|dkr tj S|kr|S|kr|S|krt ||SdS)zCompute the most frequent value in a 1d array extended with [extra_value] * n_repeat, where extra_value is assumed to be not part of the array.rc3.K|]\}}|k |VdSN).0valuecountmost_frequent_counts r% z!_most_frequent..5s?&& E5///////&&r'N) sizerobjectr most_commonminitemsrnpnan)array extra_valuen_repeatcountermost_frequent_valuemoder0s @r%_most_frequentr?)s#  zA~~ ;& ennG")"5"5a"8"8";A"> "%&&&&$+MMOO&&&##   <`. .. versionadded:: 0.20 `SimpleImputer` replaces the previous `sklearn.preprocessing.Imputer` estimator which is now removed. Parameters ---------- missing_values : int, float, str, np.nan, None or pandas.NA, default=np.nan The placeholder for the missing values. All occurrences of `missing_values` will be imputed. For pandas' dataframes with nullable integer dtypes with missing values, `missing_values` can be set to either `np.nan` or `pd.NA`. strategy : str or Callable, default='mean' The imputation strategy. - If "mean", then replace missing values using the mean along each column. Can only be used with numeric data. - If "median", then replace missing values using the median along each column. Can only be used with numeric data. - If "most_frequent", then replace missing using the most frequent value along each column. Can be used with strings or numeric data. If there is more than one such value, only the smallest is returned. - If "constant", then replace missing values with fill_value. Can be used with strings or numeric data. - If an instance of Callable, then replace missing values using the scalar statistic returned by running the callable over a dense 1d array containing non-missing values of each column. .. versionadded:: 0.20 strategy="constant" for fixed value imputation. .. versionadded:: 1.5 strategy=callable for custom value imputation. fill_value : str or numerical value, default=None When strategy == "constant", `fill_value` is used to replace all occurrences of missing_values. For string or object data types, `fill_value` must be a string. If `None`, `fill_value` will be 0 when imputing numerical data and "missing_value" for strings or object data types. copy : bool, default=True If True, a copy of X will be created. If False, imputation will be done in-place whenever possible. Note that, in the following cases, a new copy will always be made, even if `copy=False`: - If `X` is not an array of floating values; - If `X` is encoded as a CSR matrix; - If `add_indicator=True`. add_indicator : bool, default=False If True, a :class:`MissingIndicator` transform will stack onto output of the imputer's transform. This allows a predictive estimator to account for missingness despite imputation. If a feature has no missing values at fit/train time, the feature won't appear on the missing indicator even if there are missing values at transform/test time. keep_empty_features : bool, default=False If True, features that consist exclusively of missing values when `fit` is called are returned in results when `transform` is called. The imputed value is always `0` except when `strategy="constant"` in which case `fill_value` will be used instead. .. versionadded:: 1.2 .. versionchanged:: 1.6 Currently, when `keep_empty_feature=False` and `strategy="constant"`, empty features are not dropped. This behaviour will change in version 1.8. Set `keep_empty_feature=True` to preserve this behaviour. Attributes ---------- statistics_ : array of shape (n_features,) The imputation fill value for each feature. Computing statistics can result in `np.nan` values. During :meth:`transform`, features corresponding to `np.nan` statistics will be discarded. indicator_ : :class:`~sklearn.impute.MissingIndicator` Indicator used to add binary indicators for missing values. `None` if `add_indicator=False`. n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 See Also -------- IterativeImputer : Multivariate imputer that estimates values to impute for each feature with missing values from all the others. KNNImputer : Multivariate imputer that estimates missing features using nearest samples. Notes ----- Columns which only contained missing values at :meth:`fit` are discarded upon :meth:`transform` if strategy is not `"constant"`. In a prediction context, simple imputation usually performs poorly when associated with a weak learner. However, with a powerful learner, it can lead to as good or better performance than complex imputation such as :class:`~sklearn.impute.IterativeImputer` or :class:`~sklearn.impute.KNNImputer`. Examples -------- >>> import numpy as np >>> from sklearn.impute import SimpleImputer >>> imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean') >>> imp_mean.fit([[7, 2, 3], [4, np.nan, 6], [10, 5, 9]]) SimpleImputer() >>> X = [[np.nan, 2, 3], [4, np.nan, 6], [10, np.nan, 9]] >>> print(imp_mean.transform(X)) [[ 7. 2. 3. ] [ 4. 3.5 6. ] [10. 3.5 9. ]] For a more detailed example see :ref:`sphx_glr_auto_examples_impute_plot_missing_values.py`. >meanmedianconstant most_frequent no_validationrB)strategy fill_valuecopyrFrvNTF)r$r{r|r}rDrEczt|||||_||_||_dS)NrC)rerIr{r|r})rHr$r{r|r}rDrErks r%rIzSimpleImputer.__init__,sJ )' 3    ! $ r'c 0|jdvr9t|tr!td|Drt}n d}nt }|s|jjdkr|j}t|j st|j rd}nd} t|||d||sdnd||j }nP#t$rC}dt|vr+td |j|}|d|d}~wwxYw|r |j|_t#||j |jjd vr'td |jt%j|r|j d krtd |jdkr|r2|j+t+|j}d|jd|d|jd}n$|s|jj}d|d|jd}n|j}t/j||jdst||S)N)ryrxc3JK|]}|D]}t|tVdSr+)rstr)r-rowelems r%r1z0SimpleImputer._validate_input..EsQ++*-s++7; 4%%+++++++r'O allow-nanTcsc)reset accept_sparserforce_writeableensure_all_finiter}zcould not convertz0Cannot use {} strategy with non-numeric data: {}rrrrzSimpleImputer does not support data with dtype {0}. Please provide either a numeric array (with a floating point or integer dtype) or categorical data represented either as an array with integer dtype or an array of string values with an object dtype.rdImputation not possible when missing_values == 0 and input is sparse. Provide a dense array instead.rxz fill_value=z (of type z+) cannot be cast to the input data that is z2. Make sure that both dtypes are of the same kind.z%The dtype of the filling value (i.e. z\. Make sure that the dtypes of the input data is of the same kind between fit and transform. same_kind)casting)r{rlistanyr3r _fit_dtyperr r$r rr}r rr!rr&rWrXr|r" statistics_r7can_cast) rHr#in_fitrrvenew_vefill_value_dtypeerr_msgs r%_validate_inputzSimpleImputer._validate_input?s =9 9 9 !T"" s++12+++((  E $$/.#55OE + , , % d>Q0R0R % +   $  #,2 <"3Y   AA   "c"gg--#GNN r $&   &gDOAt2333 7<3 3 3( )/qw   ;q>> d1Q66!  =J & & +$/5#'#8#8 =$/==?O==@A===  +#'#3#9 )C::C?prefer_skip_nested_validationc>||d}|j|jjdvrd}n d}n|j}t j|r(|||j|j||_ n'| ||j|j||_ |S)aFit the imputer on `X`. Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) Input data, where `n_samples` is the number of samples and `n_features` is the number of features. y : Ignored Not used, present here for API consistency by convention. Returns ------- self : object Fitted estimator. TrN)rrrr missing_value) rr|rrrWrX _sparse_fitr{r$r _dense_fit)rHr#yr|s r%fitzSimpleImputer.fits$  4 0 0 ? "w|.. , J ;q>> #//4=$"5z  D   $4=$"5z  D  r'ct||j}|jdtj|jz }tj|jd}|dkrk|jsMtfdtjdDrtj dt| |nlt|jdD]P}|j|j||j|dz} ||j||j|dz} | | } t| d} | | } | } ||| z} t| dkr |jrd||<|dkr7| j| z}|dkr tjn| |z ||<|dkrt%| | ||<|dkrt'| d| ||<#t)|t*r|| ||<Rt/|S) z#Fit the transformer on sparse data.rr)rxcLg|] }tdd|fj!Sr+)alldata)r-r missing_masks r% z-SimpleImputer._sparse_fit..s1TTT!\!!!Q$',--TTTr'Currently, when `keep_empty_feature=False` and `strategy="constant"`, empty features are not dropped. This behaviour will change in version 1.8. Set `keep_empty_feature=True` to preserve this behaviour.rvrwry)r rshaper7diffindptremptyrErrangewarningswarn FutureWarningfillsumlenr2r8rr?rrr{rerQ)rHr#r{r$r| mask_datan_implicit_zeros statisticsrcolumn mask_column mask_zerosn_explicit_zerosn_zerossrrks @r%rzSimpleImputer._sparse_fitse N33  % 71:(9(99Xagaj)) z ! !+ TTTTu\=OPQ=R7S7STTT11  L"  OOJ ' ' ' '171:&& > > ahq1uo =>' ahq1uo(EF  -'vq11  ,#->>#3#3 *1-0@@v;;!##(@#$%JqMM6))"K'123q&&fjjllQ>N 1 !X--(3FG(D(D 1 !_44(6vq'(J(J 1 #Hh77>(, f(=(= 1  |,,,r'ct||}tj||}t||dkrwt j|d}t j|}|jrdn t j |t j |<|S|dkrwt j |d} t j| } |jrdn t j | t j | <| S|dkr(| }| } |jjdkr't j|jdt$} nt j|jd} t't)|d d | d d D]z\} \}}t j|t.}||}t1|dkr |jrd| | <\t3|t j d| | <{| S|d kr|jsTtj |drt9jd t<t j|jd ||jStA|tBrrt j|jd }tE|jd D]6} |#|d d | f$|| <7|Sd S) z"Fit the transformer on dense data.)maskrvraxisrwryrrNrxrr))%r ma masked_arrayrerQr7rvgetdatarEr8getmaskrw getmaskarray transposerrrrr3 enumeratezip logical_notastypeboolrr?rrrrrfullrrrr{ compressed)rHr#r{r$r|rmasked_X mean_maskedrv median_maskedrwrryrrrow_maskrrks r%rzSimpleImputer._dense_fits/ N33 ?1<888 |,,, v  %**XA*66K5==--D484L/XqqRTRXD{++ ,K ! !ELLL::MU]]=11F-926 25%%m44 5M ( (  A))++Dw|s"" "6 B B B " 4 4 &/AaaaD$qqq'0B0B&C&C F F"?C>(33::4@@(ms88q==T%=='(M!$$'5c261'E'EM!$$  # #+  80D0D0H0Ha0H0P0P0T0T0V0V  L" 7171:zAAA A( + + (."344J8>!,-- K K $ hqqq!tn.G.G.I.I J J 1     r'ct|||d}|j}|jd|jdkr/t d|jd|jjdfzt ||j}|jdks|jr|}d}nt |tj }tj |}||}tj |}| rntj|jd|}t|dr |j|}t#jd |d |jd |dd|f}t'j|r|jdkrt d ||j} nt |j|j} tjtjt/|jdz t2 tj|j| } || |jd|j| <np||} n |dd|f} tj| d} tj|| } tj| ddd}| ||<tA!|}tA"||S)ahImpute all missing values in `X`. Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) The input data to complete. Returns ------- X_imputed : {ndarray, sparse matrix} of shape (n_samples, n_features_out) `X` with imputed values. Frr)rz)X has %d features per sample, expected %drxNfeature_names_in_z/Skipping features without any observed values: zI. At least one non-missing value is needed for imputation with strategy='z'.rr)r}r)#rrrrr r r$r{rEr7r8r flatnonzerorarangerSrrrrWrXrrepeatrrintrrrrwhererrerUr\)rHr#rrvalid_statisticsvalid_statistics_indexes invalid_mask valid_maskinvalid_featuresrindexesmask_valid_features n_missingvalues coordinatesr[rks r%rTzSimpleImputer.transformOs   5 1 1% 71:)!, , ,;71:t/5a89:  !D$788  =J & &$*B &) '+ $ $%Z88L 55J)*5 ')~j'A'A $!! 3#%9QWQZ#8#8#F 4!455P'+'=>N'O$ H(HH6:mHHH aaa112 ;q>> $"a'' %,3',DD$QVT-@AAD)Ic!(mma/s;;;RWQX=N=N 08??e?TTt (/&2##&21116N3N&O#2;;;IY/;;F(#6#@#@#B#BCCDDbDIK#AkNgg22<@@ ww--a===r'c t||jstd|jdt|jj}|jd|z }|ddd|f}|dd|dft}t|j }|jd|f}tj |}||dd|jjf<|t} d\} } | t|j krZtj|dd| fs|j | |dd| f<| dz } | dz } n| dz } | t|j kZ|j|| <|S)a=Convert the data back to the original representation. Inverts the `transform` operation performed on an array. This operation can only be performed after :class:`SimpleImputer` is instantiated with `add_indicator=True`. Note that `inverse_transform` can only invert the transform in features that have binary indicators for missing values. If a feature has no missing values at `fit` time, the feature won't have a binary indicator, and the imputation done at `transform` time won't be inverted. .. versionadded:: 0.24 Parameters ---------- X : array-like of shape (n_samples, n_features + n_features_missing_indicator) The imputed data to be reverted to original data. It has to be an augmented array of imputed data and the missing indicator mask. Returns ------- X_original : ndarray of shape (n_samples, n_features) The original `X` with missing values as it was prior to imputation. zr'inverse_transform' works only when 'SimpleImputer' is instantiated with 'add_indicator=True'. Got 'add_indicator=z ' instead.r)Nr)rr)rrDr rrN features_rr}rrrr7zerosTrr$) rHr#n_features_missingnon_empty_feature_count array_imputedrn_features_originalshape_original X_original full_mask imputed_idx original_idxs r%inverse_transformzSimpleImputer.inverse_transforms8 ! '+&8 !!:;;"#'!*/A"A!!!55556;;== 34445<)D4Dbf*M*MNN/0<U>U>U>U>n<<<|TTTTTTTTr'ruceZdZUdZegeddhgdedhgdgdZeed<e j ddddd Z d Z d Z ddZedddZdZedddZddZfdZxZS)rMa Binary indicators for missing values. Note that this component typically should not be used in a vanilla :class:`~sklearn.pipeline.Pipeline` consisting of transformers and a classifier, but rather could be added using a :class:`~sklearn.pipeline.FeatureUnion` or :class:`~sklearn.compose.ColumnTransformer`. Read more in the :ref:`User Guide `. .. versionadded:: 0.20 Parameters ---------- missing_values : int, float, str, np.nan or None, default=np.nan The placeholder for the missing values. All occurrences of `missing_values` will be imputed. For pandas' dataframes with nullable integer dtypes with missing values, `missing_values` should be set to `np.nan`, since `pd.NA` will be converted to `np.nan`. features : {'missing-only', 'all'}, default='missing-only' Whether the imputer mask should represent all or a subset of features. - If `'missing-only'` (default), the imputer mask will only represent features containing missing values during fit time. - If `'all'`, the imputer mask will represent all features. sparse : bool or 'auto', default='auto' Whether the imputer mask format should be sparse or dense. - If `'auto'` (default), the imputer mask will be of same type as input. - If `True`, the imputer mask will be a sparse matrix. - If `False`, the imputer mask will be a numpy array. error_on_new : bool, default=True If `True`, :meth:`transform` will raise an error when there are features with missing values that have no missing values in :meth:`fit`. This is applicable only when `features='missing-only'`. Attributes ---------- features_ : ndarray of shape (n_missing_features,) or (n_features,) The features indices which will be returned when calling :meth:`transform`. They are computed during :meth:`fit`. If `features='all'`, `features_` is equal to `range(n_features)`. n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 See Also -------- SimpleImputer : Univariate imputation of missing values. IterativeImputer : Multivariate imputation of missing values. Examples -------- >>> import numpy as np >>> from sklearn.impute import MissingIndicator >>> X1 = np.array([[np.nan, 1, 3], ... [4, 0, np.nan], ... [8, 1, 0]]) >>> X2 = np.array([[5, 1, np.nan], ... [np.nan, 2, 3], ... [2, 4, 0]]) >>> indicator = MissingIndicator() >>> indicator.fit(X1) MissingIndicator() >>> X2_tr = indicator.transform(X2) >>> X2_tr array([[False, True], [ True, False], [False, False]]) missing-onlyrrBautor$featuresrrKrFTc>||_||_||_||_dSr+r)rHr$rrrKs r%rIzMissingIndicator.__init__cs(-   (r'c|jst||j}n|}tj|rs||jdkr|d}|jdur| }n}|j dkr| }n]|jst||j}n|}|jdkr| d}|jdurtj |}|jdkr tj|jd}ntj|}||fS) aCompute the imputer mask and the indices of the features containing missing values. Parameters ---------- X : {ndarray, sparse matrix} of shape (n_samples, n_features) The input data with missing values. Note that `X` has been checked in :meth:`fit` and :meth:`transform` before to call this function. Returns ------- imputer_mask : {ndarray, sparse matrix} of shape (n_samples, n_features) The imputer mask of the original data. features_with_missing : ndarray of shape (n_features_with_missing) The features containing missing values. rrrFcsrTrr)) _precomputedr r$rWrXeliminate_zerosrgetnnzrtoarrayr!tocscr csc_matrixr7rrr)rHr# imputer_maskrfeatures_indicess r%_get_missing_features_infoz+MissingIndicator._get_missing_features_infopsR(  $Q(;<> ;  ( ( * * *}..(//Q/77 {e##+3355 $--+1133 $ !(D,?@@  }..(,,!,44 {d""!}\:: =E ! !!y44  !~i88 ---r'cRt|jsd}nd}t|||dd|}t||j|jjdvr't d|jtj |r|jdkrt d|S) NTr)rr)rrrrrzMissingIndicator does not support data with dtype {0}. Please provide either a numeric array (with a floating point or integer dtype) or categorical data represented either as an array with integer dtype or an array of string values with an object dtype.rzSSparse input with missing_values=0 is not supported. Provide a dense array instead.) r r$rr&rrr r!rWrX)rHr#rrs r%rz MissingIndicator._validate_inputsT011 , $   +    (/     At2333 7<3 3 3( )/qw   ;q>> d1Q66!  r'NFc\|r7t|dr|jjdkstdd|_nd|_|js||d}nt ||d|jd|_| |}|d|_ |d S) aOFit the transformer on `X`. Parameters ---------- X : {array-like, sparse matrix} of shape (n_samples, n_features) Input data, where `n_samples` is the number of samples and `n_features` is the number of features. If `precomputed=True`, then `X` is a mask of the input data. precomputed : bool Whether the input data is a mask. Returns ------- imputer_mask : {ndarray, sparse matrix} of shape (n_samples, n_features) The imputer mask of the original data. rb4precomputed is True but the input data is not a maskTFr)rr)r) rSrrr rrrr _n_featuresr r)rHr#rrLmissing_features_infos r%rOzMissingIndicator._fits&  &Aw'' YAGLC,?,? !WXXX $D   %D   3$$Qt$44AA dAT 2 2 2 271: $ ? ? B B.q1$Q''r'rc2||||S)aFit the transformer on `X`. Parameters ---------- X : {array-like, sparse matrix} of shape (n_samples, n_features) Input data, where `n_samples` is the number of samples and `n_features` is the number of features. y : Ignored Not used, present for API consistency by convention. Returns ------- self : object Fitted estimator. )rO)rHr#rs r%rzMissingIndicator.fits$ !Q r'ct||js||d}n/t|dr|jjdkst d||\}}|jdkrttj ||j }|j r-|j dkr"t d||j j |jkr|d d |j f}|S) aGenerate missing values indicator for `X`. Parameters ---------- X : {array-like, sparse matrix} of shape (n_samples, n_features) The input data to complete. Returns ------- Xt : {ndarray, sparse matrix} of shape (n_samples, n_features) or (n_samples, n_features_with_missing) The missing indicator for input data. The data type of `Xt` will be boolean. FrrrrrrzSThe features {} have missing values in transform but have no missing values in fit.N)rrrrSrrr r rr7 setdiff1drrKr2r!r)rHr#r rfeatures_diff_fit_transs r%rTzMissingIndicator.transforms   Y$$Qu$55AAAw'' YAGLC,?,? !WXXX!%!@!@!C!C h =N * *&(l8T^&L&L #  %<%AA%E%E $f%<== ~"T%555+AAAt~,=> r'c~|||}|jj|jkr|dd|jf}|S)aGenerate missing values indicator for `X`. Parameters ---------- X : {array-like, sparse matrix} of shape (n_samples, n_features) The input data to complete. y : Ignored Not used, present for API consistency by convention. Returns ------- Xt : {ndarray, sparse matrix} of shape (n_samples, n_features) or (n_samples, n_features_with_missing) The missing indicator for input data. The data type of `Xt` will be boolean. N)rOrr2r)rHr#rr s r% fit_transformzMissingIndicator.fit_transform1sC&yyA > !1 1 1'4>(9:Lr'ct|dt||}|jjt jfd||jDtS)rrcg|] }d| S)_r,)r- feature_nameprefixs r%rz:MissingIndicator.get_feature_names_out..cs7    **L**   r'r) rrrkrllowerr7asarrayrr3)rHrars @r%r^z&MissingIndicator.get_feature_names_outKs( .///0~FF(..00z    $24>$B        r'ct}d|j_d|j_d|j_g|j_|Sr)rerfrgrhstringrtransformer_tagspreserves_dtyperis r%rfz!MissingIndicator.__sklearn_tags__jsFww''))$(!!%!%02- r')NFr+)rlrmrnrorrrFrprqr7r8rIr rrOr rrTrr^rfrrrss@r%rMrMsRRj)=??+Z 7889jj&223" $$Dv ) ) ) ) )4.4.4.l!!!F'('('('(R\55565*'''R\555652    >r'rM)*rr collectionsr functoolsrtypingrnumpyr7numpy.marscipyrrWbaserr r utils._maskr utils._missingr r utils._param_validationrr utils.fixesrutils.sparsefuncsrutils.validationrrrrrr&r?rArurMr,r'r%r0s@@@@@@@@@@######88888888????????++++++    "5"5"5JIIIII#]IIIXj Tj Tj Tj Tj TLj Tj Tj TZiiiii'iiiiir'