0PhdZddlZddlZddlmZmZddlmZddlmZm Z ddl Z ddl m Z mZmZddlm ZddlmZd d lmZmZmZmZd d lmZd d lmZmZd d lmZd dl m!Z!d dl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)d dl*m+Z+m,Z,m-Z-m.Z.m/Z/d dl0m1Z1m2Z2m3Z3m4Z4d dl5m6Z6m7Z7d dl8m9Z9d dl:m;Z;mZ>m?Z?d dl@mAZAd dlBmCZCmDZDmEZEddlFmGZGmHZHmIZImJZJddlKmLZLdZM dRdZNddddddddZOd ZPdSd"ZQdTd#ZR dUd$ZSd%ZTe4d&d'ejUgd&ge2e ddd()d&ge2e ddd*)d&dge3hd+ge2eddd()dge2e ddd()gd,gd-gd.gd-gd-gd-gd/ d0dd1dddd!dd!d!dd2 d3ZV dVd4ZWd5ZXd6ZYGd7d8eHe9ZZGd:d;eeeZZ[Gd<d=eGZ\Gd>d?e\eZZ]d@Z^dAZ_GdBdCej jUZ`GdDdEej jUZaGdFdGeeZbGdHdIeGeZcGdJdKeHZdGdLdMeHZeGdNdOeeeeZfGdPdQe\eeZgdS)Wz Ridge regression N)ABCMetaabstractmethod)partial)IntegralReal)linalgoptimizesparse)r) BaseEstimator)MultiOutputMixinRegressorMixin _fit_context is_classifier)ConvergenceWarning) check_scoringget_scorer_names) GridSearchCV)LabelBinarizer)Bunch check_arraycheck_consistent_length check_scalar column_or_1dcompute_sample_weight deprecated)_is_numpy_namespace_raveldevice get_namespaceget_namespace_and_device)HiddenInterval StrOptionsvalidate_params) row_normssafe_sparse_dot)_sparse_linalg_cg)MetadataRouter MethodMapping_raise_for_params_routing_enabledprocess_routing)mean_variance_axis)_check_sample_weightcheck_is_fitted validate_data)LinearClassifierMixin LinearModel_preprocess_data _rescale_data) sag_solvercvfd}fd}tjj||}|S)zCreate LinearOperator for matrix products with implicit centering. Matrix product `LinearOperator @ coef` returns `(X - X_offset) @ coef`. c`||zz SN)dotbXX_offsetsample_weight_sqrts [/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/sklearn/linear_model/_ridge.pymatvecz&_get_rescaled_operator..matvec@s(uuQxx,quuX>>>cjj||zz Sr:)Tr;r<s rArmatvecz'_get_rescaled_operator..rmatvecCs,swwqzzHquu-?'@'@@@@rC)shaperBrF)r rLinearOperatorrG)r>r?r@rBrFX1s``` rA_get_rescaled_operatorrJ:sw ???????AAAAAAA  % %AGFG % T TB IrC-C6?c j|&tj|jd|j}|j\} } ||t j|n||z } t || |tj|jd| f|j} | | krfd} nfd} t|jdD]}|dd|f}| ||}| | krLt j | | f||j}t|||\}} || |<nL |}t j | | f||j}t||||\| |<}|dkrtd |z|%|dkr|rtjd |zt| S) Nrdtyper2cfd}|S)Nc`||zzSr:)rBrFxrI curr_alphas rA_mvz0_solve_sparse_cg..create_mv.._mves(yyA//*q.@@rCrSrTrIs` rA create_mvz#_solve_sparse_cg..create_mvd/ A A A A A AJrCcfd}|S)Nc`||zzSr:)rFrBrQs rArTz0_solve_sparse_cg..create_mv.._mvms(zz"))A,,//*q.@@rCrUrVs` rArWz#_solve_sparse_cg..create_mvlrXrC)rBrN)rtol)maxiterr[zFailed with error code %dz/sparse_cg did not converge after %d iterations.)nponesrGrN sp_linalgaslinearoperatorrJemptyrangerHr(rF ValueErrorwarningswarnr)r>yalphamax_itertolverboser?X_scaler@ n_samples n_featuresX_offset_scalecoefsrWiy_columnmvCcoefinforIs @rA_solve_sparse_cgrvJs$!WQWQZqw???GIz7?  ' * *!G+ #A~7I J J Hagaj*-QW = = =EI            171:  QQQT7 YuQx   ! !(I&rA+1hSAAAJD$zz$''E!HHzz(++H(Z(17A/q(HSVWWWNE!Hd !8884?@@ @  qW MADH"    LrCT) fit_interceptrhrir?rkr@c X|&tj|jd|j}t j|r|r||z } t || |} n|} |j\} } tj|jd| f|j} tj|jdtj}tj |}t|jdD]D}|dd|f}tj | ||||||}|d| |<|d||<E| |fS)aSolve Ridge regression via LSQR. We expect that y is always mean centered. If X is dense, we expect it to be mean centered such that we can solve ||y - Xw||_2^2 + alpha * ||w||_2^2 If X is sparse, we expect X_offset to be given such that we can solve ||y - (X - X_offset)w||_2^2 + alpha * ||w||_2^2 With sample weights S=diag(sample_weight), this becomes ||sqrt(S) (y - (X - X_offset) w)||_2^2 + alpha * ||w||_2^2 and we expect y and X to already be rescaled, i.e. sqrt(S) @ y, sqrt(S) @ X. In this case, X_offset is the sample_weight weighted mean of X before scaling by sqrt(S). The objective then reads ||y - (X - sqrt(S) X_offset) w)||_2^2 + alpha * ||w||_2^2 NrrMr2)dampatolbtoliter_limr ) r]r^rGrNr issparserJraint32sqrtrbr_lsqr)r>rfrgrwrhrir?rkr@rnrIrlrmron_iter sqrt_alpharprqrus rA _solve_lsqrrs:8!WQWQZqw??? qm!G+ #A~7I J JGIz Hagaj*-QW = = =E Xagaj 1 1 1FJ 171:  QQQT7~ z!}3S8   7aGq &=rCc|jd}|jd}t|j|d}t|j|d}tj|t ||dgz}|r>|jdd|dzxx|dz cc<tj||ddjStj ||g|j }t||j|D]k\} } } |jdd|dzxx| z cc<tj|| dd | dd<|jdd|dzxx| zcc<l|S) Nr2T dense_outputrposassume_a overwrite_arMF) rGr'rEr] array_equallenflatrsolverarNzipravel) r>rfrgrm n_targetsAXy one_alpharorttarget current_alphas rA_solve_choleskyrsJ IQT222A ad 3 3 3Buc%jjE!H:&=>>I    *q. !!!U1X-!!!|ArEtDDDFF)Z0@@@+.ubdE+B+B 7 7 'D&- F$$j1n$ % % % 6 % % %l1fu%PPPVVXXDG F$$j1n$ % % % 6 % % % % rCFc|jd}|jd}|r|}tj|}||dk}t |tjp|dv}|rWtjtj|} || ddtjfz}|tj | | z}|r|j dd|dzxx|dz cc< tj ||dd} nI#tj j $r2tjdtj||d} YnwxYw|j dd|dzxx|dzcc<|r| | ddtjfz} | Stj||g|j} t'| |j|D]k\} } } |j dd|dzxx| z cc<tj || dd| dd<|j dd|dzxx| zcc<l|r| | tjddfz} | jS)Nrr2)?NrFrzNSingular matrix in solving dual problem. Using least-squares solution instead.)rGcopyr] atleast_1dall isinstancendarrayrnewaxisouterrrr LinAlgErrorrdrelstsqrarNrrEr)Krfrg sample_weightrrlrrhas_swsw dual_coef dual_coefsrrs rA_solve_cholesky_kernelrs I I  FFHH M% E%("''))I  rz 2 2 Vm;6VF WR]=11 2 2 111bj=! ! RXb"  ( )a-   E!H,    . QEuMMMIIy$ . . . M2    Q**1-III  . )a-   E!H,    + AAArzM* *IXy)4ag>> 03JU0K0K 6 6 ,Iv} F##i!m# $ $ $ 5 $ $ $!<6Euegg aaaL F##i!m# $ $ $ 5 $ $ $ $  , "RZ]+ +J|s>DAEEcvt||\}}|j|d\}}}|dk}||dddf} |j|z} ||jd|jdf|jt|} | | dz|zz | |<| | z} |j| zjS)NxpF full_matricesgV瞯rfrgr_UsVtidxs_nnzUTydd_UT_ys rA _solve_svdrs ! # # #EBy}}Qe}44HAq" e)C cF111d7OE #'C !'!*ek!n-QWVAYYOOA eQh& 'AcF WF D6M rCc Hj\} } i} ||| d<d|d| d} |rdtjfg| z| d<||||z nd&tjjdjtj|jd | fj} t |jd D]stj| f}|ddffd }tj ||fi| }|d s tj d |d |t|d| <t| S)a4Solve ridge regression with LBFGS. The main purpose is fitting with forcing coefficients to be positive. For unconstrained ridge regression, there are faster dedicated solver methods. Note that with positive bounds on the coefficients, LBFGS seems faster than scipy.optimize.lsq_linear. Nr\zL-BFGS-BT)methodrijacoptionsrboundsrMr2cP| z }||zz}d||zdz||zz}j|z|zz}||zz}||fS)Ng?)r;rE) wresidualfgradr>rnrgrpr@rqs rAfuncz_solve_lbfgs..funcYsuuQxx(*H).~1F1FFFhll8,,,sU1X~a/HHA3>E!HqL0D)6H)I)IIId7NrCsuccesszWThe lbfgs solver did not converge. Try increasing max_iter or tol. Currently: max_iter=z and tol=rR) rGr]infr^rNrarbrr minimizerdrer)r>rfrgpositiverhrir?rkr@rlrmrconfigrox0rresultrnrprqs` ` ` @@@rA _solve_lbfgsr+s$GIzG%  F 6K=:5x 3!G+!WQWQZqw??? Hagaj*-QW = = =E 171:   Xzm $ $QQQT7          "466v66i   ML3;LLFILL#    #;a LrCc|r|dvrdSgdS)N)autosagsagacsrrcsccoorU) is_X_sparsesolvers rA_get_valid_accept_sparserrs&%v!888u$$$$rC array-likez sparse matrixleftclosedneither>rrrrrlbfgscholesky sparse_cgrjboolean random_state r>rfrgrrrhrirjrr return_n_iterreturn_intercept check_inputprefer_skip_nested_validationr) rrrhrirjrrrrrc >t|||||||||| | | dd| S)aSolve the ridge equation by the method of normal equations. Read more in the :ref:`User Guide `. Parameters ---------- X : {array-like, sparse matrix, LinearOperator} of shape (n_samples, n_features) Training data. y : array-like of shape (n_samples,) or (n_samples, n_targets) Target values. alpha : float or array-like of shape (n_targets,) Constant that multiplies the L2 term, controlling regularization strength. `alpha` must be a non-negative float i.e. in `[0, inf)`. When `alpha = 0`, the objective is equivalent to ordinary least squares, solved by the :class:`LinearRegression` object. For numerical reasons, using `alpha = 0` with the `Ridge` object is not advised. Instead, you should use the :class:`LinearRegression` object. If an array is passed, penalties are assumed to be specific to the targets. Hence they must correspond in number. sample_weight : float or array-like of shape (n_samples,), default=None Individual weights for each sample. If given a float, every sample will have the same weight. If sample_weight is not None and solver='auto', the solver will be set to 'cholesky'. .. versionadded:: 0.17 solver : {'auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga', 'lbfgs'}, default='auto' Solver to use in the computational routines: - 'auto' chooses the solver automatically based on the type of data. - 'svd' uses a Singular Value Decomposition of X to compute the Ridge coefficients. It is the most stable solver, in particular more stable for singular matrices than 'cholesky' at the cost of being slower. - 'cholesky' uses the standard scipy.linalg.solve function to obtain a closed-form solution via a Cholesky decomposition of dot(X.T, X) - 'sparse_cg' uses the conjugate gradient solver as found in scipy.sparse.linalg.cg. As an iterative algorithm, this solver is more appropriate than 'cholesky' for large-scale data (possibility to set `tol` and `max_iter`). - 'lsqr' uses the dedicated regularized least-squares routine scipy.sparse.linalg.lsqr. It is the fastest and uses an iterative procedure. - 'sag' uses a Stochastic Average Gradient descent, and 'saga' uses its improved, unbiased version named SAGA. Both methods also use an iterative procedure, and are often faster than other solvers when both n_samples and n_features are large. Note that 'sag' and 'saga' fast convergence is only guaranteed on features with approximately the same scale. You can preprocess the data with a scaler from sklearn.preprocessing. - 'lbfgs' uses L-BFGS-B algorithm implemented in `scipy.optimize.minimize`. It can be used only when `positive` is True. All solvers except 'svd' support both dense and sparse data. However, only 'lsqr', 'sag', 'sparse_cg', and 'lbfgs' support sparse input when `fit_intercept` is True. .. versionadded:: 0.17 Stochastic Average Gradient descent solver. .. versionadded:: 0.19 SAGA solver. max_iter : int, default=None Maximum number of iterations for conjugate gradient solver. For the 'sparse_cg' and 'lsqr' solvers, the default value is determined by scipy.sparse.linalg. For 'sag' and saga solver, the default value is 1000. For 'lbfgs' solver, the default value is 15000. tol : float, default=1e-4 Precision of the solution. Note that `tol` has no effect for solvers 'svd' and 'cholesky'. .. versionchanged:: 1.2 Default value changed from 1e-3 to 1e-4 for consistency with other linear models. verbose : int, default=0 Verbosity level. Setting verbose > 0 will display additional information depending on the solver used. positive : bool, default=False When set to ``True``, forces the coefficients to be positive. Only 'lbfgs' solver is supported in this case. random_state : int, RandomState instance, default=None Used when ``solver`` == 'sag' or 'saga' to shuffle the data. See :term:`Glossary ` for details. return_n_iter : bool, default=False If True, the method also returns `n_iter`, the actual number of iteration performed by the solver. .. versionadded:: 0.17 return_intercept : bool, default=False If True and if X is sparse, the method also returns the intercept, and the solver is automatically changed to 'sag'. This is only a temporary fix for fitting the intercept with sparse data. For dense data, use sklearn.linear_model._preprocess_data before your regression. .. versionadded:: 0.17 check_input : bool, default=True If False, the input arrays X and y will not be checked. .. versionadded:: 0.21 Returns ------- coef : ndarray of shape (n_features,) or (n_targets, n_features) Weight vector(s). n_iter : int, optional The actual number of iteration performed by the solver. Only returned if `return_n_iter` is True. intercept : float or ndarray of shape (n_targets,) The intercept of the model. Only returned if `return_intercept` is True and if X is a scipy sparse array. Notes ----- This function won't compute the intercept. Regularization improves the conditioning of the problem and reduces the variance of the estimates. Larger values specify stronger regularization. Alpha corresponds to ``1 / (2C)`` in other linear models such as :class:`~sklearn.linear_model.LogisticRegression` or :class:`~sklearn.svm.LinearSVC`. If an array is passed, penalties are assumed to be specific to the targets. Hence they must correspond in number. Examples -------- >>> import numpy as np >>> from sklearn.datasets import make_regression >>> from sklearn.linear_model import ridge_regression >>> rng = np.random.RandomState(0) >>> X = rng.randn(100, 4) >>> y = 2.0 * X[:, 0] - 1.0 * X[:, 1] + 0.1 * rng.standard_normal(100) >>> coef, intercept = ridge_regression(X, y, alpha=1.0, return_intercept=True) >>> list(coef) [np.float64(1.9...), np.float64(-1.0...), np.float64(-0.0...), np.float64(-0.0...)] >>> intercept np.float64(-0.0...) N) rrrhrirjrrrrrkr?r)_ridge_regressionrs rAridge_regressionrysIT    # !#)   rCct t|||| |\}}}t|}tj|}|du}t ||| ||}|r|st j|}|s!|dkrtd|jd|d|r|dkrtd|d|dkr|std| r|d krtd |rI|j |j g}t||}t|||d }t||j d d}t|||j\}}|jdkr$tdt#|jz|jdkr||d}|j\}}||krtd||fz|r0t'|||j }|dvrt)|||\}}}|Ot+|t-|dgst/|dt0jdd}t5||||j |}|jdd|fvrtd|jd|fz|jddkr+|dkr%||f|d|j |}d}|dkrt9|||||||| |r|nd }nR|d!kr!t;|||||||| |r|nd" \}}n+|d#kr||kr_t=||jd$%} tA| ||}!t=|j|!d$%j}n#tBj"$rd}YnwxYw tG|||}n#tBj"$rd}YnwxYw|dvrmtI|d$&%}"t j&|jd|f|j }t j&|jdt j'}t j(|jdf|j }#tStU||jD]\}$\}%}&d't j(|tW| zdf|j i}'tY||&-|d(|%d|||| d |"|'|d)k*\}(})}*| r|(dd+||$<|(d+|#|$<n|(||$<|)||$<|#jddkr|#d}#n"|dkrt]|||||||| |r|nd, }|dkr#|rt_d-ta||||}|dkrt5|}||}| r| r|||#f}+n| r||#f}+n | r||f}+n|}+| rg|+|Rn|+S).Nrz Array API dispatch to namespace z" only supports solver 'svd'. Got 'z'.rzJWhen positive=True, only 'lbfgs' solver can be used. Please change solver z" to 'lbfgs' or set positive=False.N'lbfgs' solver can be used only when positive=True. Please use another solver.rzxIn Ridge, only 'sag' solver can directly fit the intercept. Please change solver to 'sag' or set return_intercept=False.rs) accept_sparserNorderF)rN ensure_2drr zTarget y has the wrong shape %sr2)r2z:Number of samples in X and y does not correspond: %d != %drM)rrrgr target_typemin_valinclude_boundaries)rrNrrzENumber of targets and number of penalties do not correspond: %d != %d)rG fill_valuerNrr)rhrirjr?rkr@r)rgrwrhrir?rkr@rTr)squaredrtrr)is_sagar)rrirhr?rkr@z3SVD solver does not support sparse inputs currently)1r!rr r}resolve_solverr]asarrayrc__name__float64float32rrrNrrGndimstrreshaper/r6rtypernumbersrrfullrvrr'rErrrrr&maxrar~r enumeraterintr7rr TypeErrorr),r>rfrgrrrhrirjrrrr return_solverrkr?rrwris_array_api_compliantdevice_is_numpy_namespace X_is_sparser_dtype_accept_sparserlrm n_samples_rr@rrtrrmax_squared_sum interceptrpalpha_irinitcoef_n_iter_rress, rArrVs}&+C 1mWh++'B-R00/!$$K $ &F FH.> R P PF+ JqMM  &E// -r{ - -"( - - -    Fg%% %$* % % %    )    FeOO &   G*bj)1+vFF vS Q Q Q E F F FAq!!!GIzvzz:S\\IJJJv{{ JJq' " "GJ J H*% &   J,]AQWMMM  ( ((5Q='I'I $Aq$ E4 C58I8I3J3J!K!K   %     2::eG17:CC K K KE {1~a^++ S{1~y) *    {1~y1}},585;w  F  5;E11     6  " '5;E11    ff :    ! !13T:::A 21a?? &qsIDIIIK%     &q!U33%     ? " "#At44488::xZ0@@@!'!*BH555Hagaj]!'::: $-c%oo$>$>  A *s3C/D/D"Da!HPQPWXXXD!+ &(!!! E7A   *Q$Ry ! QF1II ?1  " "!! I 7   5;E11      SQRR R!Qr**A~~d|| ::d  D)FI% Io Fl* 3>S>&>>>3s$;-M**M?>M?NN+*N+c |dkr|St|}t|||}|r|S|rtd|jdd}||kr&t jd|jd|d|d|d |S) NrzYThe solvers that support positive fitting do not support Array API dispatch to namespace zc. Please either disable Array API dispatch, or use a numpy-like namespace, or set `positive=False`.rz&Using Array API dispatch to namespace z7 with `solver='auto'` will result in using the solver 'zr'. The results may differ from those when using a Numpy array, because in that case the preferred solver would be z. Set `solver='z'` to suppress this warning.)rresolve_solver_for_numpyrcrrdre)rrr is_sparserr auto_solver_nps rArr;s  ,R00-h8H)TTN  2/1{ 2 2 2   F   AR[ A A@F A ACQ A A#  A A A    MrCc|rdS|rdS|sdSdS)NrrrrrU)rrrs rArr[s.wu z ;rCc eZdZUeedddejgdgdgeeddddgeedddgehdgdgdgd Z e e d <e dd d dd dddddZ ddZdS) _BaseRidgerNrrrr2>rrrrrrrrrrgrwcopy_Xrhrirrr_parameter_constraintsrTrKrFrwr rhrirrrcv||_||_||_||_||_||_||_||_dSr:r) selfrgrwr rhrirrrs rA__init__z_BaseRidge.__init__ysD *      (rCct|||\}}|jdkr|jstd|jr)|jdvrtd|jd|j}nt j|r|jrx|jdvr'td|j|jdvr|j}n>|jd kr)|j"|j d krtj d d }n d }n|j}|t|||j }t|||j|j|\}}}}} |d krzt j|rf|jr_t!|||j||j|j d |j|jdddd \|_|_|_|_|xj|z c_nt j|r |jr|| d} ni} t!||f|j||j|j ||j|jdddd|jd | \|_|_|_|||| |S)Nrr)rrzsolver='zh' does not support positive fitting. Please set the solver to 'auto' or 'lbfgs', or set `positive=False`)rrrrrzsolver='{}' does not support fitting the intercept on sparse data. Please set the solver to 'auto' or 'lsqr', 'sparse_cg', 'sag', 'lbfgs' or set `fit_intercept=False`)rrrrKz"sag" solver requires many iterations to fit an intercept with sparse inputs. Either set the solver to "auto" or "sparse_cg", or set a low "tol" and a high "max_iter" (especially if inputs are not standardized).rrMrwrrTF) rgrrhrirrrrrr r)r?rk) rgrrhrirrrrrr rrw)r rrrcr r}rwformatrhrirdrer/rNr5r rrgrrr intercept_solver__set_intercept) r$r>rfrrr rr?y_offsetrkparamss rAfitz_BaseRidge.fits%21a%G%G" " ;' ! !$- !-  = !{"333 Tt{TTT  _Q   !D$6 !{"OOO 34:6$+3F3F  {///%%$-*?DHtOO )$[F  $0QQQM-= ,' - - - )1h' U??vq11?d6H?FWj+H!."!%"!GGG CDJ dot| OOx 'OOOq!! d&8 &.7CC5F6j+H!."!&"!"06666 2DJ dl"   (G < < < rCrr:)r __module__ __qualname__r#rr]rrr$r!dict__annotations__rr%r.rUrCrArris(4D888"*E#+Xh4???Fq$v6667 JXXX   K'( $ $D   ) ))))^)*ccccccrCr) metaclassc reZdZdZ d ddddddddfd Zed dfd Zfd ZxZS)RidgeaRLinear least squares with l2 regularization. Minimizes the objective function:: ||y - Xw||^2_2 + alpha * ||w||^2_2 This model solves a regression model where the loss function is the linear least squares function and regularization is given by the l2-norm. Also known as Ridge Regression or Tikhonov regularization. This estimator has built-in support for multi-variate regression (i.e., when y is a 2d-array of shape (n_samples, n_targets)). Read more in the :ref:`User Guide `. Parameters ---------- alpha : {float, ndarray of shape (n_targets,)}, default=1.0 Constant that multiplies the L2 term, controlling regularization strength. `alpha` must be a non-negative float i.e. in `[0, inf)`. When `alpha = 0`, the objective is equivalent to ordinary least squares, solved by the :class:`LinearRegression` object. For numerical reasons, using `alpha = 0` with the `Ridge` object is not advised. Instead, you should use the :class:`LinearRegression` object. If an array is passed, penalties are assumed to be specific to the targets. Hence they must correspond in number. fit_intercept : bool, default=True Whether to fit the intercept for this model. If set to false, no intercept will be used in calculations (i.e. ``X`` and ``y`` are expected to be centered). copy_X : bool, default=True If True, X will be copied; else, it may be overwritten. max_iter : int, default=None Maximum number of iterations for conjugate gradient solver. For 'sparse_cg' and 'lsqr' solvers, the default value is determined by scipy.sparse.linalg. For 'sag' solver, the default value is 1000. For 'lbfgs' solver, the default value is 15000. tol : float, default=1e-4 The precision of the solution (`coef_`) is determined by `tol` which specifies a different convergence criterion for each solver: - 'svd': `tol` has no impact. - 'cholesky': `tol` has no impact. - 'sparse_cg': norm of residuals smaller than `tol`. - 'lsqr': `tol` is set as atol and btol of scipy.sparse.linalg.lsqr, which control the norm of the residual vector in terms of the norms of matrix and coefficients. - 'sag' and 'saga': relative change of coef smaller than `tol`. - 'lbfgs': maximum of the absolute (projected) gradient=max|residuals| smaller than `tol`. .. versionchanged:: 1.2 Default value changed from 1e-3 to 1e-4 for consistency with other linear models. solver : {'auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga', 'lbfgs'}, default='auto' Solver to use in the computational routines: - 'auto' chooses the solver automatically based on the type of data. - 'svd' uses a Singular Value Decomposition of X to compute the Ridge coefficients. It is the most stable solver, in particular more stable for singular matrices than 'cholesky' at the cost of being slower. - 'cholesky' uses the standard scipy.linalg.solve function to obtain a closed-form solution. - 'sparse_cg' uses the conjugate gradient solver as found in scipy.sparse.linalg.cg. As an iterative algorithm, this solver is more appropriate than 'cholesky' for large-scale data (possibility to set `tol` and `max_iter`). - 'lsqr' uses the dedicated regularized least-squares routine scipy.sparse.linalg.lsqr. It is the fastest and uses an iterative procedure. - 'sag' uses a Stochastic Average Gradient descent, and 'saga' uses its improved, unbiased version named SAGA. Both methods also use an iterative procedure, and are often faster than other solvers when both n_samples and n_features are large. Note that 'sag' and 'saga' fast convergence is only guaranteed on features with approximately the same scale. You can preprocess the data with a scaler from sklearn.preprocessing. - 'lbfgs' uses L-BFGS-B algorithm implemented in `scipy.optimize.minimize`. It can be used only when `positive` is True. All solvers except 'svd' support both dense and sparse data. However, only 'lsqr', 'sag', 'sparse_cg', and 'lbfgs' support sparse input when `fit_intercept` is True. .. versionadded:: 0.17 Stochastic Average Gradient descent solver. .. versionadded:: 0.19 SAGA solver. positive : bool, default=False When set to ``True``, forces the coefficients to be positive. Only 'lbfgs' solver is supported in this case. random_state : int, RandomState instance, default=None Used when ``solver`` == 'sag' or 'saga' to shuffle the data. See :term:`Glossary ` for details. .. versionadded:: 0.17 `random_state` to support Stochastic Average Gradient. Attributes ---------- coef_ : ndarray of shape (n_features,) or (n_targets, n_features) Weight vector(s). intercept_ : float or ndarray of shape (n_targets,) Independent term in decision function. Set to 0.0 if ``fit_intercept = False``. n_iter_ : None or ndarray of shape (n_targets,) Actual number of iterations for each target. Available only for sag and lsqr solvers. Other solvers will return None. .. versionadded:: 0.17 n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 solver_ : str The solver that was used at fit time by the computational routines. .. versionadded:: 1.5 See Also -------- RidgeClassifier : Ridge classifier. RidgeCV : Ridge regression with built-in cross validation. :class:`~sklearn.kernel_ridge.KernelRidge` : Kernel ridge regression combines ridge regression with the kernel trick. Notes ----- Regularization improves the conditioning of the problem and reduces the variance of the estimates. Larger values specify stronger regularization. Alpha corresponds to ``1 / (2C)`` in other linear models such as :class:`~sklearn.linear_model.LogisticRegression` or :class:`~sklearn.svm.LinearSVC`. Examples -------- >>> from sklearn.linear_model import Ridge >>> import numpy as np >>> n_samples, n_features = 10, 5 >>> rng = np.random.RandomState(0) >>> y = rng.randn(n_samples) >>> X = rng.randn(n_samples, n_features) >>> clf = Ridge(alpha=1.0) >>> clf.fit(X, y) Ridge() rTNrKrFr"c Zt||||||||dSNr)superr%) r$rgrwr rhrirrr __class__s rAr%zRidge.__init__sH '%  rCrc ttj||j}t |||\}}t |||||j|jgddd\}}t |||S)a%Fit Ridge regression model. Parameters ---------- X : {ndarray, sparse matrix} of shape (n_samples, n_features) Training data. y : ndarray of shape (n_samples,) or (n_samples, n_targets) Target values. sample_weight : float or ndarray of shape (n_samples,), default=None Individual weights for each sample. If given a float, every sample will have the same weight. Returns ------- self : object Fitted estimator. T)rrNforce_writeable multi_output y_numericr) rr r}rr r1rrr9r.)r$r>rfrrrrr:s rAr.z Ridge.fits*2&/!2D2DdkRRaM22A  (:rz*     1ww{{1a}{===rCct}d|_|jdko|jdkp|j |j_|S)NTrr)r9__sklearn_tags__array_api_supportrrw input_tagsr r$tagsr:s rArAzRidge.__sklearn_tags__sUww''))!%"&+"6" K: % ?T-?)?  rCr/r:) rr0r1__doc__r%rr.rA __classcell__r:s@rAr6r6sqqj         .\555 > > > > >65 >DrCr6cDeZdZdZfdZedZfdZxZS)_RidgeClassifierMixinc ttj||}t||||ddd\}}t dd|_|j|}|jjdst|d}t|||j }|j r|t|j |z}||||fS) aValidate `X` and `y` and binarize `y`. Parameters ---------- X : {ndarray, sparse matrix} of shape (n_samples, n_features) Training data. y : ndarray of shape (n_samples,) Target values. sample_weight : float or ndarray of shape (n_samples,), default=None Individual weights for each sample. If given a float, every sample will have the same weight. solver : str The solver used in `Ridge` to know which sparse format to support. Returns ------- X : {ndarray, sparse matrix} of shape (n_samples, n_features) Validated training data. y : ndarray of shape (n_samples,) Validated target values. sample_weight : ndarray of shape (n_samples,) Validated sample weights. Y : ndarray of shape (n_samples, n_classes) The binarized version of `y`. TF)rr=r>r<r2r) pos_label neg_label multilabel)rerM)rr r}r1r_label_binarizer fit_transformy_type_ startswithrr/rN class_weightr)r$r>rfrrrYs rA _prepare_dataz#_RidgeClassifierMixin._prepare_datas@11C1CVLL   '    1!/b I I I  ! / / 2 2$,77 EE +QT***A,]AQWMMM   X),A$BSUV,W,WWM!]A%%rCct|dg|jjdr9d||dkzdz }|j|St |S)aIPredict class labels for samples in `X`. Parameters ---------- X : {array-like, spare matrix} of shape (n_samples, n_features) The data matrix for which we want to predict the targets. Returns ------- y_pred : ndarray of shape (n_samples,) or (n_samples, n_outputs) Vector or matrix containing the predictions. In binary and multiclass problems, this is a vector containing `n_samples`. In a multilabel problem, it returns a matrix of shape `(n_samples, n_outputs)`. rO) attributesrNr rr2)r0rOrQrRdecision_functioninverse_transformr9predict)r$r>scoresr:s rArZz_RidgeClassifierMixin.predict"s *<)=>>>>  ( 3 3L A A C$0033a781`. Parameters ---------- alpha : float, default=1.0 Regularization strength; must be a positive float. Regularization improves the conditioning of the problem and reduces the variance of the estimates. Larger values specify stronger regularization. Alpha corresponds to ``1 / (2C)`` in other linear models such as :class:`~sklearn.linear_model.LogisticRegression` or :class:`~sklearn.svm.LinearSVC`. fit_intercept : bool, default=True Whether to calculate the intercept for this model. If set to false, no intercept will be used in calculations (e.g. data is expected to be already centered). copy_X : bool, default=True If True, X will be copied; else, it may be overwritten. max_iter : int, default=None Maximum number of iterations for conjugate gradient solver. The default value is determined by scipy.sparse.linalg. tol : float, default=1e-4 The precision of the solution (`coef_`) is determined by `tol` which specifies a different convergence criterion for each solver: - 'svd': `tol` has no impact. - 'cholesky': `tol` has no impact. - 'sparse_cg': norm of residuals smaller than `tol`. - 'lsqr': `tol` is set as atol and btol of scipy.sparse.linalg.lsqr, which control the norm of the residual vector in terms of the norms of matrix and coefficients. - 'sag' and 'saga': relative change of coef smaller than `tol`. - 'lbfgs': maximum of the absolute (projected) gradient=max|residuals| smaller than `tol`. .. versionchanged:: 1.2 Default value changed from 1e-3 to 1e-4 for consistency with other linear models. class_weight : dict or 'balanced', default=None Weights associated with classes in the form ``{class_label: weight}``. If not given, all classes are supposed to have weight one. The "balanced" mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as ``n_samples / (n_classes * np.bincount(y))``. solver : {'auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga', 'lbfgs'}, default='auto' Solver to use in the computational routines: - 'auto' chooses the solver automatically based on the type of data. - 'svd' uses a Singular Value Decomposition of X to compute the Ridge coefficients. It is the most stable solver, in particular more stable for singular matrices than 'cholesky' at the cost of being slower. - 'cholesky' uses the standard scipy.linalg.solve function to obtain a closed-form solution. - 'sparse_cg' uses the conjugate gradient solver as found in scipy.sparse.linalg.cg. As an iterative algorithm, this solver is more appropriate than 'cholesky' for large-scale data (possibility to set `tol` and `max_iter`). - 'lsqr' uses the dedicated regularized least-squares routine scipy.sparse.linalg.lsqr. It is the fastest and uses an iterative procedure. - 'sag' uses a Stochastic Average Gradient descent, and 'saga' uses its unbiased and more flexible version named SAGA. Both methods use an iterative procedure, and are often faster than other solvers when both n_samples and n_features are large. Note that 'sag' and 'saga' fast convergence is only guaranteed on features with approximately the same scale. You can preprocess the data with a scaler from sklearn.preprocessing. .. versionadded:: 0.17 Stochastic Average Gradient descent solver. .. versionadded:: 0.19 SAGA solver. - 'lbfgs' uses L-BFGS-B algorithm implemented in `scipy.optimize.minimize`. It can be used only when `positive` is True. positive : bool, default=False When set to ``True``, forces the coefficients to be positive. Only 'lbfgs' solver is supported in this case. random_state : int, RandomState instance, default=None Used when ``solver`` == 'sag' or 'saga' to shuffle the data. See :term:`Glossary ` for details. Attributes ---------- coef_ : ndarray of shape (1, n_features) or (n_classes, n_features) Coefficient of the features in the decision function. ``coef_`` is of shape (1, n_features) when the given problem is binary. intercept_ : float or ndarray of shape (n_targets,) Independent term in decision function. Set to 0.0 if ``fit_intercept = False``. n_iter_ : None or ndarray of shape (n_targets,) Actual number of iterations for each target. Available only for sag and lsqr solvers. Other solvers will return None. classes_ : ndarray of shape (n_classes,) The classes labels. n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 solver_ : str The solver that was used at fit time by the computational routines. .. versionadded:: 1.5 See Also -------- Ridge : Ridge regression. RidgeClassifierCV : Ridge classifier with built-in cross validation. Notes ----- For multi-class classification, n_class classifiers are trained in a one-versus-all approach. Concretely, this is implemented by taking advantage of the multi-variate response support in Ridge. Examples -------- >>> from sklearn.datasets import load_breast_cancer >>> from sklearn.linear_model import RidgeClassifier >>> X, y = load_breast_cancer(return_X_y=True) >>> clf = RidgeClassifier().fit(X, y) >>> clf.score(X, y) 0.9595... rSbalancedNr!rTrKrF)rwr rhrirSrrrc ht|||||||| ||_dSr8r9r%rS) r$rgrwr rhrirSrrrr:s rAr%zRidgeClassifier.__init__sN '%  )rCrc|||||j\}}}}t||||S)asFit Ridge classifier model. Parameters ---------- X : {ndarray, sparse matrix} of shape (n_samples, n_features) Training data. y : ndarray of shape (n_samples,) Target values. sample_weight : float or ndarray of shape (n_samples,), default=None Individual weights for each sample. If given a float, every sample will have the same weight. .. versionadded:: 0.17 *sample_weight* support to RidgeClassifier. Returns ------- self : object Instance of the estimator. r?)rUrr9r.)r$r>rfrrTr:s rAr.zRidgeClassifier.fit sJ0"&!3!3Aq-!U!U1mQ  Aq 666 rCct}|jdko|jdkp|j |j_|S)Nrr)r9rArrwrCr rDs rArAz RidgeClassifier.__sklearn_tags__&sMww''))"&+"6" K: % ?T-?)?  rCr/r:)rr0r1rFrr!r2r$r3r%rr.rArGrHs@rArereFs bbH$  +$zz:,77>$$D) )))))))2\555658rCrecN|dvr|S|jd|jdkrdSdS)N)eigenrrr2rrlrG)r>gcv_modes rA_check_gcv_modero.s6### wqzAGAJu 7rCc|tj||}tj|}|S)aFind the column of vectors that is most aligned with the query. Both query and the columns of vectors must have their l2 norm equal to 1. Parameters ---------- query : ndarray of shape (n_samples,) Normalized query vector. vectors : ndarray of shape (n_samples, n_features) Vectors to which we compare query, as columns. Must be normalized. )r]absr;argmax)queryvectors abs_cosineindexs rA_find_smallest_anglerw8s3 '**++J Ij ! !E LrCc4eZdZdZfdZdZdZdZxZS)_X_CenterStackOpzBehaves as centered and scaled X with an added intercept column. This operator behaves as np.hstack([X - sqrt_sw[:, None] * X_mean, sqrt_sw[:, None]]) c|j\}}t|j||dzf||_||_||_dSNr2rGr9r%rNr>X_meansqrt_swr$r>r}r~rlrmr:s rAr%z_X_CenterStackOp.__init__QsM ! : 9j1n"=>>>  rCc|}t|j|ddd|j|j|ddzz |d|jzzSNrTr)rr'r>r~r}r;r$vs rA_matvecz_X_CenterStackOp._matvecXsh GGII DFAcrcF > > >lT[__QssV444 5edl" # rCct|j|ddd|jdddf|j|ddzz |d|jdddfzzSr)r'r>r~r}r;rs rA_matmatz_X_CenterStackOp._matmat`su DFAcrcF > > >l111d7#dkooaf&=&== >edl111d7++ , rCcBt|j|j|jSr:)_XT_CenterStackOpr>r}r~r^s rA _transposez_X_CenterStackOp._transposegs dlCCCrC) rr0r1rFr%rrrrGrHs@rAryryJsv       DDDDDDDrCryc.eZdZdZfdZdZdZxZS)rzBehaves as transposed centered and scaled X with an intercept column. This operator behaves as np.hstack([X - sqrt_sw[:, None] * X_mean, sqrt_sw[:, None]]).T c|j\}}t|j|dz|f||_||_||_dSr{r|rs rAr%z_XT_CenterStackOp.__init__rsM ! : :>9"=>>>  rCcJ|}|jd}tj||jj}t |jj|d|j|j |zz |dd<tj ||j |d<|S)NrrMTrr) rrGr]rar>rNr'rEr}r~r;r$rrmrs rArz_XT_CenterStackOp._matvecys GGIIZ] hz666"468QTBBB K$,**1-- - CRC&DL))B rCcP|jd}tj||jdf|jj}t |jj|d|jdddf|j |zz |dd<tj |j||d<|S)Nrr2rMTrr) rGr]rar>rNr'rEr}r~r;rs rArz_XT_CenterStackOp._matmatsZ] h AGAJ/tv|DDD"468QTBBBT[ AAtGF L  Q  F  CRC&q))B rC)rr0r1rFr%rrrGrHs@rArrks` rCrceZdZdZdZdZdS)_IdentityRegressorz9Fake regressor which will directly output the prediction.c|Sr:rUr$ y_predicts rArXz$_IdentityRegressor.decision_functionrCc|Sr:rUrs rArZz_IdentityRegressor.predictrrCN)rr0r1rFrXrZrUrCrArrs8CCrCrceZdZdZdZdZdS)_IdentityClassifierzFake classifier which will directly output the prediction. We inherit from LinearClassifierMixin to get the proper shape for the output `y`. c||_dSr:)r])r$classess rAr%z_IdentityClassifier.__init__s  rCc|Sr:rUrs rArXz%_IdentityClassifier.decision_functionrrCN)rr0r1rFr%rXrUrCrArrs<    rCrc eZdZdZ ddddddddddZedZed Zd Zd Z d Z d Z dZ dZ dZdZdZdZdZddZdZdZfdZxZS) _RidgeGCVa Ridge regression with built-in Leave-one-out Cross-Validation. This class is not intended to be used directly. Use RidgeCV instead. `_RidgeGCV` uses a Generalized Cross-Validation for model selection. It's an efficient approximation of leave-one-out cross-validation (LOO-CV), where instead of computing multiple models by excluding one data point at a time, it uses an algebraic shortcut to approximate the LOO-CV error, making it faster and computationally more efficient. Using a naive grid-search approach with a leave-one-out cross-validation in contrast requires to fit `n_samples` models to compute the prediction error for each sample and then to repeat this process for each alpha in the grid. Here, the prediction error for each sample is computed by solving a **single** linear system (in other words a single model) via a matrix factorization (i.e. eigendecomposition or SVD) solving the problem stated in the Notes section. Finally, we need to repeat this process for each alpha in the grid. The detailed complexity is further discussed in Sect. 4 in [1]. This algebraic approach is only applicable for regularized least squares problems. It could potentially be extended to kernel ridge regression. See the Notes section and references for more details regarding the formulation and the linear system that is solved. Notes ----- We want to solve (K + alpha*Id)c = y, where K = X X^T is the kernel matrix. Let G = (K + alpha*Id). Dual solution: c = G^-1y Primal solution: w = X^T c Compute eigendecomposition K = Q V Q^T. Then G^-1 = Q (V + alpha*Id)^-1 Q^T, where (V + alpha*Id) is diagonal. It is thus inexpensive to inverse for many alphas. Let loov be the vector of prediction values for each example when the model was fitted with all examples but this example. loov = (KG^-1Y - diag(KG^-1)Y) / diag(I-KG^-1) Let looe be the vector of prediction errors for each example when the model was fitted with all examples but this example. looe = y - loov = c / diag(G^-1) The best score (negative mean squared error or user-provided scoring) is stored in the `best_score_` attribute, and the selected hyperparameter in `alpha_`. References ---------- [1] http://cbcl.mit.edu/publications/ps/MIT-CSAIL-TR-2007-025.pdf [2] https://www.mit.edu/~9.520/spring07/Classes/rlsslides.pdf g?rg$@TNF)rwscoringr rnstore_cv_resultsis_clfalpha_per_targetcv||_||_||_||_||_||_||_||_dSr:)alphasrwrr rnrrr) r$rrwrr rnrrrs rAr%z_RidgeGCV.__init__sE *    0 0rCc:||dzzdS)Nr raxis)sum)v_primeQs rA _decomp_diagz_RidgeGCV._decomp_diags#!Q$###,,,rCct|jdkr<|tdftjft|jdz zz}||zSr{)rrGslicer]r)DBs rA _diag_dotz_RidgeGCV._diag_dotsM qw<r~centerr}rlsample_weight_matrix X_weightedrX_mXX_mX_ms rA _compute_gramz_RidgeGCV._compute_gramsQ>#:(:(: FXagaj888F"1ac===vE EGAJ %0 aLI 6   *--a00 &z::: )gkk'2222qqq$w/&!#D"Q"Q"QQ'7++bfVV.D.DD Aqs 6 6 6 ?$ F O   rCc|js?tj|jd|j}t |j|d|fS|jd}tj|df||f}| |}t|d\}}||z| |z }| |}t |j|d|tj ||zz |fS)a\Computes covariance matrix X^TX with possible centering. Parameters ---------- X : sparse matrix of shape (n_samples, n_features) The preprocessed design matrix. sqrt_sw : ndarray of shape (n_samples,) square roots of sample weights Returns ------- covariance : ndarray of shape (n_features, n_features) The covariance matrix. X_mean : ndarray of shape (n_feature,) The weighted mean of ``X`` for each feature. Notes ----- Since X is sparse it has not been centered in preprocessing, but it has been scaled by sqrt(sample weights). When self.fit_intercept is False no centering is done. The centered X is never actually computed because centering would break the sparsity of X. r2rMTrrrmr) rwr]rrGrNr'rEr rr;r.r) r$r>r~r}rlrrr weight_sums rA_compute_covariancez_RidgeGCV._compute_covariance:s 8! FXagaj888F"13===vE EGAJ %0 aLI 6   *--a00 &z::: )#gkk'&:&::[[)) AC 6 6 628FF333 4   rCc |x}}|jd}tj|jd|j}t d|jd|D] } t | t |jd| |zd} tj|| jd|jd|jzf|j} |jrI|| ||| dddfzz | ddddf<|| | dddf<n|| } | || z d|| < |S)a9Compute the diagonal of (X - X_mean).dot(A).dot((X - X_mean).T) without explicitly centering X nor computing X.dot(A) when X is sparse. Parameters ---------- X : sparse matrix of shape (n_samples, n_features) A : ndarray of shape (n_features, n_features) X_mean : ndarray of shape (n_features,) sqrt_sw : ndarray of shape (n_features,) square roots of sample weights Returns ------- diag : np.ndarray, shape (n_samples,) The computed diagonal. r2rrMNrr) rGr]rarNrbrminrwtoarrayr;r) r$r>rr}r~ intercept_colscale batch_sizediagstartbatchX_batchs rA_sparse_multidot_diagz_RidgeGCV._sparse_multidot_diagjsd*!(' WQZ x !'2221agaj*55 A AE%QWQZ1C!D!DaHHEh5"AGAJ1C$CDAGG! -"#E("2"2"4"4ve QQQPTW@U7U"U3B3!.u!52E(**,,";;q>>G388a8@@DKK rCc|||\}}|jr|tj||z }t j|\}}tj|j|}||||fS)z?Eigendecomposition of X.X^T, used when n_samples <= n_features.)rrwr]rreighr;rE) r$r>rfr~rr}eigvalsrQT_ys rA_eigen_decompose_gramz_RidgeGCV._eigen_decompose_gramsq&&q'22 6   , '7++ +A[^^ vac1~~w4''rCcpd||zz }|jr7|tj|z } t | |} d|| <tj||||} |||} t|j dkr| ddtj f} | | fS)zCompute dual coefficients and diagonal of G^-1. Used when we have a decomposition of X.X^T (n_samples <= n_features). rrr2N rwr]rnormrwr;rrrrGr) r$rgrfr~r}rrrr normalized_sw intercept_dimcG_inverse_diags rA_solve_eigen_gramz_RidgeGCV._solve_eigen_grams 7U? #   ! $binnW&=&==M0BBM Am  F1dnnQ-- . .**1a00 qw<<1  +AAArzM:Nq  rCc|j\}}tj|dz|dzf|j}|||\|ddddf<}|js|ddddf}n&d|d<d|dddf<|||d<td||z }tj |\} } | |d} | dd|df} || | |fS)z_Eigendecomposition of X^T.X, used when n_samples > n_features and X is sparse. r2rMNrr)rr) rGr]rarNrrwr;rrr) r$r>rfr~rlrmcovr} nullspace_dimrVs rA_eigen_decompose_covariancez%_RidgeGCV._eigen_decompose_covariances !" :h Q Q7qwGGG $ 8 8G D DCRC"H v! /crc3B3h-CCCGC2J!++g..CKAzI566 [%% -..) aaa w1$$rCcpd||zz }||z|j} | t|j|d} t|| d} ||| ||} t |jdkr| ddt jf} d| z |z || z |z fS)zCompute dual coefficients and diagonal of G^-1. Used when we have a decomposition of X^T.X (n_samples > n_features and X is sparse), and not fitting an intercept. r2TrN)r;rEr'rrrGr]r) r$rgrfr~r}rrr>rrAXyy_hathat_diags rA$_solve_eigen_covariance_no_interceptz._RidgeGCV._solve_eigen_covariance_no_intercepts 5 ! UKK  eeOAC>>>??3T:::--aFGDD qw<<1  2: .HH %E U':::rCc"tj|jd}d|d<t||} d||zz } d|| z | | <|| z|j} t |||} | | j|} | | }||| ||}t|jdkr|ddtj f}d|z |z ||z |z fS)zCompute dual coefficients and diagonal of G^-1. Used when we have a decomposition of X^T.X (n_samples > n_features and X is sparse), and we are fitting an intercept. rr2rN) r]rrGrwr;rEryrrr)r$rgrfr~r}rrr> intercept_svrrrX_oprrrs rA!_solve_eigen_covariance_interceptz+_RidgeGCV._solve_eigen_covariance_interceptsx ++  R,\1== 5 !w}55- UKK  6733eeDFJJqMM"" --aFGDD qw<<1  2: .HH %E U':::rCc ||jr||||||||S||||||||S)zCompute dual coefficients and diagonal of G^-1. Used when we have a decomposition of X^T.X (n_samples > n_features and X is sparse). )rwrr)r$rgrfr~r}rrr>s rA_solve_eigen_covariancez!_RidgeGCV._solve_eigen_covariances_   99q'67Aq 88 1gvw1   rCctj|jd|j}|jr"|dddf}tj||f}t j|d\}}}|dz} tj|j |} || || fS)Nr2rMrrr ) r]rrGrNrwhstackrrr;rE) r$r>rfr~r}intercept_columnrsingvalsr singvals_sqUT_ys rA_svd_decompose_design_matrixz&_RidgeGCV._svd_decompose_design_matrix s!'!*AG444   1 'qqq$w/  1./00AAQ7778Qk vac1~~{At++rCc||zdz|dzz }|jr;|tj|z } t | |} |dz || <tj|||||dz|zz} ||||dzz} t|j dkr| ddtj f} | | fS)zCompute dual coefficients and diagonal of G^-1. Used when we have an SVD decomposition of X (n_samples > n_features and X is dense). rr2Nr) r$rgrfr~r}rrrrrrrrs rA_solve_svd_design_matrixz"_RidgeGCV._solve_svd_design_matrixs E!b (UBY 7   ,#binnW&=&==M0BBM!&|Am  F1dnnQ-- . .%)q @**1a00E2I> qw<<1  +AAArzM:Nq  rCc t|||gdtjgdd\}}|jr |jrJ|t |||j}tj|j|_|}t|||j |j |\}}}}}t||j } | dkr|j} |j} n7| dkr1t!j|r|j} |j} n|j} |j} |jd } |t/|||\}}} ntj| |j} | ||| ^}}t3|jd krd n |jd }tj|jd krd nt3|j}|jr%tj| |z|f|j|_d \}}}t=tj|jD]p\}}| tA||| |g|R\}}|j!F||z d z}|"| }|jr |#|jdd|f<ny|||z z }| |jd kr|| dddfz}n|| z}||z }|jr |#|jdd|f<|pi}|$||||j!|}|B|jr3|d kr-|}tj|}tj%||}$|}|}|},|jr1|d kr+||k}|dd|f|dd|f<||||<|||<d||kr|||}}}r||_&||_'||_(tS|j(j*||_+|jd ks|jd d kr|j+#|_+t!j|r||z}n|||zz }|,||||jrAt3|jd kr| |f}n| ||f}|j-||_|S)aFit Ridge regression model with gcv. Parameters ---------- X : {ndarray, sparse matrix} of shape (n_samples, n_features) Training data. Will be cast to float64 if necessary. y : ndarray of shape (n_samples,) or (n_samples, n_targets) Target values. Will be cast to float64 if necessary. sample_weight : float or ndarray of shape (n_samples,), default=None Individual weights for each sample. If given a float, every sample will have the same weight. Note that the scale of `sample_weight` has an impact on the loss; i.e. multiplying all weights by `k` is equivalent to setting `alpha / k`. score_params : dict, default=None Parameters to be passed to the underlying scorer. .. versionadded:: 1.5 See :ref:`Metadata Routing User Guide ` for more details. Returns ------- self : object rT)rrNr=r>NrMr'rlrrr2)NNNr )squared_errors) predictionsrfn_yscorer score_params).r1r]rrrr/rNrrr5rwr rornrrr r}rrrrrGr6r^rrrra cv_results_rrfloatr_score_without_scorerr_scoreralpha_ best_score_ dual_coef_r'rErr+r)r$r>rfrr unscaled_yr?r,rkrn decomposerrlr~r} decompositionrn_alphas best_coef best_score best_alpharprgrrr alpha_scorer to_updatecv_results_shapes rAr.z _RidgeGCV.fit.s8  ///:,   1K:D$9:::  $0QQQMj--  ,< ,' - - - )1h'#1dm44 w  2I*EE   q!! 6 < 4 = 5GAJ  $)!Q >>MAq''giqw777G!*1a!9!9qw<<1$$aa!'!* ,,1111s4;7G7G   T!xS((C17SSSD ,<) :z!"- "<"<==0 N0 NHAu %eEllAw W W W W NA|#"#n"4!:"888WW (D-;-A-A-C-CD$QQQT*1~#56  ,"'!++#wqqq$w'77 #w. x' (A-8->->-@-@D$QQQT*+1r "kk + Performs scoring using squared errors when the scorer is None.rr)rmean)r$rrs rArz_RidgeGCV._score_without_scorersA   ,$))q)111FF$))+++F rCcF|jrBttj|dfi}nRt |jr2tjfdt|D}n fi}|S)zhPerforms scoring with the specified scorer using the predictions and the true y values. )rr2rc Ng|]!}dd|fdd|ffi"Sr:rU).0jidentity_estimatorrrrrfs rA z$_RidgeGCV._score..sh .'1-aaadG+ rC) rrr]arangerrrrarrayrb)r$rrfrrrrr s `` `` @rArz_RidgeGCV._scores ; T!4RYs^^!L!L!L V"a   FF"4!5!5 $ T"'s    2KSSlSS rCc`t}d|j_|Sr`)r9rA target_tagsrequiredrDs rArAz_RidgeGCV.__sklearn_tags__s(ww''))$(! rCr)NN)rr0r1rFr% staticmethodrrrrrrrrrrrrrr.rrrArGrHs@rArrs<<@ 111111*--\-\2 2 2 h. . . `###J ( ( (!!!.%%%2;;;$;;;>     , , ,!!!(]]]]~>rCrc `eZdZUdeedddgdgeeeedgdgehddgde dgdgde ed hgd Z e e d < dd dddddd ddZ ddZdZdZededZfdZxZS) _BaseRidgeCVrrNrrr cv_object>rrrlrrrwrcvrnrrstore_cv_valuesr!rTF)rwrrrnrrrcv||_||_||_||_||_||_||_||_dSr:r) r$rrwrrrnrrrs rAr%z_BaseRidgeCV.__init__ sF *   0 0.rCc Lt||d|j}|}|jdkr=|jt dt jdt|j|_ n|jd|_ n |j|_ |#tttj dd }n"tttj dd }t|jt jt$t&frt!j|jd krd nt+|j}|d kr+t-|jD]\} } || d | d} n||jd d|jd <t!j|j} |||d<|t1rt3|dfi|} n/t5t5i} ||| jjd<|jd}t=| |j||j |j tC||j"} | #|||| jj| j$|_$| j%|_%|j r | j&|_&n|j rt d|j"rt dd| i}tOj(|rdnd}tC|rtRntT}||j|} t1r| +dtY| |||}|j#||fi||j-} |j-j.|_$|j%|_%| j/|_/| j0|_0| j1|_1te| dr | j3|_3|S)aFit Ridge regression model with cv. Parameters ---------- X : ndarray of shape (n_samples, n_features) Training data. If using GCV, will be cast to float64 if necessary. y : ndarray of shape (n_samples,) or (n_samples, n_targets) Target values. Will be cast to X's dtype if necessary. sample_weight : float or ndarray of shape (n_samples,), default=None Individual weights for each sample. If given a float, every sample will have the same weight. **params : dict, default=None Extra parameters for the underlying scorer. .. versionadded:: 1.5 Only available if `enable_metadata_routing=True`, which can be set by using ``sklearn.set_config(enable_metadata_routing=True)``. See :ref:`Metadata Routing User Guide ` for more details. Returns ------- self : object Fitted estimator. Notes ----- When sample_weight is provided, the selected hyperparameter may depend on whether we use leave-one-out cross-validation (cv=None or cv='auto') or another form of cross-validation, because only leave-one-out cross-validation takes the sample weights into account when computing the validation score. r.rNzBoth 'store_cv_values' and 'store_cv_results' were set. 'store_cv_values' is deprecated in version 1.5 and will be removed in 1.7. To avoid this error, only set 'store_cv_results'.zj'store_cv_values' is deprecated in version 1.5 and will be removed in 1.7. Use 'store_cv_results' instead.Frrrrrr2zalphas[]rr)score)r)rwrrnrrr)rrz3cv!=None and store_cv_results=True are incompatiblez3cv!=None and alpha_per_target=True are incompatiblergrr)rwrTr?)rrfeature_names_in_)4r+r _get_scorerrrrcrdre FutureWarning_store_cv_resultsrrrrrrr]rlisttuplerrrrr,r-rrrrrrwrnrrr.rrrr r}rer6set_fit_requestrbest_estimator_rgrr)n_features_in_hasattrr)r$r>rfrr-rrcheck_scalar_alpharrvrgr routed_params estimator parametersrmodel grid_searchs rAr.z_BaseRidgeCV.fit s@N &$... W!!##  < / /$0 X MF    &*%9D " "  " *%*D " "%)%:D " :!(#L#, """  ")#L#) """  dkBJe#< = = NGDK00A55qq3t{;K;KH1}}$-dk$:$:JJLE5..u6H6H6H6HIIEEJ"4!3DKNH!M!M ADK((  $&3F? # :!! P /!!!! !&U___ = = =  ,BOM(.?|#!"0!%!7$T**!%!6I MM+*17     $*DK(4D % 9#,#8 % X !VWWW$ X !VWWW!6*J$*OA$6$6B[[FF'4T':':EOOE"0I !! >)))===& K KOAq + +F + + +#3I%5;DK*6D _ #.'6 91 2 2 A%.%@D " rCcLt|jj||jt dd|jt dd}|S)ajGet metadata routing of this object. Please check :ref:`User Guide ` on how the routing mechanism works. .. versionadded:: 1.5 Returns ------- routing : MetadataRouter A :class:`~sklearn.utils.metadata_routing.MetadataRouter` encapsulating routing information. )ownerr.r)callercallee)rmethod_mappingsplit)splitterr0)r)r:radd_self_requestaddrr*r)r$routers rAget_metadata_routingz!_BaseRidgeCV.get_metadata_routing s !8 9 9 9  d # # S|,22%2PPS,22%2PP  rCct||jd}tr|j|d|S)NT)r(r allow_noner?)rrr,set_score_request)r$rs rArz_BaseRidgeCV._get_scorer sLt|PTUUU    9$,"6  $ $4 $ 8 8 8 rCzjAttribute `cv_values_` is deprecated in version 1.5 and will be removed in 1.7. Use `cv_results_` instead.c|jSr:)rr^s rA cv_values_z_BaseRidgeCV.cv_values_ s rCc`t}d|j_|Sr`)r9rArCr rDs rArAz_BaseRidgeCV.__sklearn_tags__ s'ww''))!% rCrr:)rr0r1r#rr$setrcallabler"r!r2r3r%r.r6rrrcr;rArGrHs@rArrs$4 !J!J!JK#Jss#3#3#5#566774HmZ 8 8 8994@&t 5&K%vvjj,.H.H'I'IJ $ $D    / $/////*bbbbH8Z -  X   rCrcHeZdZdZeddfd ZxZS)RidgeCVaRidge regression with built-in cross-validation. See glossary entry for :term:`cross-validation estimator`. By default, it performs efficient Leave-One-Out Cross-Validation. Read more in the :ref:`User Guide `. Parameters ---------- alphas : array-like of shape (n_alphas,), default=(0.1, 1.0, 10.0) Array of alpha values to try. Regularization strength; must be a positive float. Regularization improves the conditioning of the problem and reduces the variance of the estimates. Larger values specify stronger regularization. Alpha corresponds to ``1 / (2C)`` in other linear models such as :class:`~sklearn.linear_model.LogisticRegression` or :class:`~sklearn.svm.LinearSVC`. If using Leave-One-Out cross-validation, alphas must be strictly positive. fit_intercept : bool, default=True Whether to calculate the intercept for this model. If set to false, no intercept will be used in calculations (i.e. data is expected to be centered). scoring : str, callable, default=None A string (see :ref:`scoring_parameter`) or a scorer callable object / function with signature ``scorer(estimator, X, y)``. If None, the negative mean squared error if cv is 'auto' or None (i.e. when using leave-one-out cross-validation), and r2 score otherwise. cv : int, cross-validation generator or an iterable, default=None Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the efficient Leave-One-Out cross-validation - integer, to specify the number of folds. - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs, if ``y`` is binary or multiclass, :class:`~sklearn.model_selection.StratifiedKFold` is used, else, :class:`~sklearn.model_selection.KFold` is used. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. gcv_mode : {'auto', 'svd', 'eigen'}, default='auto' Flag indicating which strategy to use when performing Leave-One-Out Cross-Validation. Options are:: 'auto' : use 'svd' if n_samples > n_features, otherwise use 'eigen' 'svd' : force use of singular value decomposition of X when X is dense, eigenvalue decomposition of X^T.X when X is sparse. 'eigen' : force computation via eigendecomposition of X.X^T The 'auto' mode is the default and is intended to pick the cheaper option of the two depending on the shape of the training data. store_cv_results : bool, default=False Flag indicating if the cross-validation values corresponding to each alpha should be stored in the ``cv_results_`` attribute (see below). This flag is only compatible with ``cv=None`` (i.e. using Leave-One-Out Cross-Validation). .. versionchanged:: 1.5 Parameter name changed from `store_cv_values` to `store_cv_results`. alpha_per_target : bool, default=False Flag indicating whether to optimize the alpha value (picked from the `alphas` parameter list) for each target separately (for multi-output settings: multiple prediction targets). When set to `True`, after fitting, the `alpha_` attribute will contain a value for each target. When set to `False`, a single alpha is used for all targets. .. versionadded:: 0.24 store_cv_values : bool Flag indicating if the cross-validation values corresponding to each alpha should be stored in the ``cv_values_`` attribute (see below). This flag is only compatible with ``cv=None`` (i.e. using Leave-One-Out Cross-Validation). .. deprecated:: 1.5 `store_cv_values` is deprecated in version 1.5 in favor of `store_cv_results` and will be removed in version 1.7. Attributes ---------- cv_results_ : ndarray of shape (n_samples, n_alphas) or shape (n_samples, n_targets, n_alphas), optional Cross-validation values for each alpha (only available if ``store_cv_results=True`` and ``cv=None``). After ``fit()`` has been called, this attribute will contain the mean squared errors if `scoring is None` otherwise it will contain standardized per point prediction values. .. versionchanged:: 1.5 `cv_values_` changed to `cv_results_`. coef_ : ndarray of shape (n_features) or (n_targets, n_features) Weight vector(s). intercept_ : float or ndarray of shape (n_targets,) Independent term in decision function. Set to 0.0 if ``fit_intercept = False``. alpha_ : float or ndarray of shape (n_targets,) Estimated regularization parameter, or, if ``alpha_per_target=True``, the estimated regularization parameter for each target. best_score_ : float or ndarray of shape (n_targets,) Score of base estimator with best alpha, or, if ``alpha_per_target=True``, a score for each target. .. versionadded:: 0.23 n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 See Also -------- Ridge : Ridge regression. RidgeClassifier : Classifier based on ridge regression on {-1, 1} labels. RidgeClassifierCV : Ridge classifier with built-in cross validation. Examples -------- >>> from sklearn.datasets import load_diabetes >>> from sklearn.linear_model import RidgeCV >>> X, y = load_diabetes(return_X_y=True) >>> clf = RidgeCV(alphas=[1e-3, 1e-2, 1e-1, 1]).fit(X, y) >>> clf.score(X, y) 0.5166... TrNc Btj||fd|i||S)aFit Ridge regression model with cv. Parameters ---------- X : ndarray of shape (n_samples, n_features) Training data. If using GCV, will be cast to float64 if necessary. y : ndarray of shape (n_samples,) or (n_samples, n_targets) Target values. Will be cast to X's dtype if necessary. sample_weight : float or ndarray of shape (n_samples,), default=None Individual weights for each sample. If given a float, every sample will have the same weight. **params : dict, default=None Parameters to be passed to the underlying scorer. .. versionadded:: 1.5 Only available if `enable_metadata_routing=True`, which can be set by using ``sklearn.set_config(enable_metadata_routing=True)``. See :ref:`Metadata Routing User Guide ` for more details. Returns ------- self : object Fitted estimator. Notes ----- When sample_weight is provided, the selected hyperparameter may depend on whether we use leave-one-out cross-validation (cv=None or cv='auto') or another form of cross-validation, because only leave-one-out cross-validation takes the sample weights into account when computing the validation score. r)r9r.)r$r>rfrr-r:s rAr.z RidgeCV.fit s0P  Aq@@ @@@@ rCr:)rr0r1rFrr.rGrHs@rAr@r@ sdNN`\555(((((65(((((rCr@ceZdZUdZiejdeedhdgiZeed<dD]Z e e  ddddddd d fd Z e d dfd Z xZS)RidgeClassifierCVaRidge classifier with built-in cross-validation. See glossary entry for :term:`cross-validation estimator`. By default, it performs Leave-One-Out Cross-Validation. Currently, only the n_features > n_samples case is handled efficiently. Read more in the :ref:`User Guide `. Parameters ---------- alphas : array-like of shape (n_alphas,), default=(0.1, 1.0, 10.0) Array of alpha values to try. Regularization strength; must be a positive float. Regularization improves the conditioning of the problem and reduces the variance of the estimates. Larger values specify stronger regularization. Alpha corresponds to ``1 / (2C)`` in other linear models such as :class:`~sklearn.linear_model.LogisticRegression` or :class:`~sklearn.svm.LinearSVC`. If using Leave-One-Out cross-validation, alphas must be strictly positive. fit_intercept : bool, default=True Whether to calculate the intercept for this model. If set to false, no intercept will be used in calculations (i.e. data is expected to be centered). scoring : str, callable, default=None A string (see :ref:`scoring_parameter`) or a scorer callable object / function with signature ``scorer(estimator, X, y)``. cv : int, cross-validation generator or an iterable, default=None Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the efficient Leave-One-Out cross-validation - integer, to specify the number of folds. - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. class_weight : dict or 'balanced', default=None Weights associated with classes in the form ``{class_label: weight}``. If not given, all classes are supposed to have weight one. The "balanced" mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as ``n_samples / (n_classes * np.bincount(y))``. store_cv_results : bool, default=False Flag indicating if the cross-validation results corresponding to each alpha should be stored in the ``cv_results_`` attribute (see below). This flag is only compatible with ``cv=None`` (i.e. using Leave-One-Out Cross-Validation). .. versionchanged:: 1.5 Parameter name changed from `store_cv_values` to `store_cv_results`. store_cv_values : bool Flag indicating if the cross-validation values corresponding to each alpha should be stored in the ``cv_values_`` attribute (see below). This flag is only compatible with ``cv=None`` (i.e. using Leave-One-Out Cross-Validation). .. deprecated:: 1.5 `store_cv_values` is deprecated in version 1.5 in favor of `store_cv_results` and will be removed in version 1.7. Attributes ---------- cv_results_ : ndarray of shape (n_samples, n_targets, n_alphas), optional Cross-validation results for each alpha (only if ``store_cv_results=True`` and ``cv=None``). After ``fit()`` has been called, this attribute will contain the mean squared errors if `scoring is None` otherwise it will contain standardized per point prediction values. .. versionchanged:: 1.5 `cv_values_` changed to `cv_results_`. coef_ : ndarray of shape (1, n_features) or (n_targets, n_features) Coefficient of the features in the decision function. ``coef_`` is of shape (1, n_features) when the given problem is binary. intercept_ : float or ndarray of shape (n_targets,) Independent term in decision function. Set to 0.0 if ``fit_intercept = False``. alpha_ : float Estimated regularization parameter. best_score_ : float Score of base estimator with best alpha. .. versionadded:: 0.23 classes_ : ndarray of shape (n_classes,) The classes labels. n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 See Also -------- Ridge : Ridge regression. RidgeClassifier : Ridge classifier. RidgeCV : Ridge regression with built-in cross validation. Notes ----- For multi-class classification, n_class classifiers are trained in a one-versus-all approach. Concretely, this is implemented by taking advantage of the multi-variate response support in Ridge. Examples -------- >>> from sklearn.datasets import load_breast_cancer >>> from sklearn.linear_model import RidgeClassifierCV >>> X, y = load_breast_cancer(return_X_y=True) >>> clf = RidgeClassifierCV(alphas=[1e-3, 1e-2, 1e-1, 1]).fit(X, y) >>> clf.score(X, y) 0.9630... rSrfNr!)rnrrTr)rwrrrSrrcdt||||||||_dS)N)rrwrrrrrh) r$rrwrrrSrrr:s rAr%zRidgeClassifierCV.__init__A sH '-+    )rCrc ||||d\}}}}|j|n|}tj||fd|i||S)arFit Ridge classifier with cv. Parameters ---------- X : ndarray of shape (n_samples, n_features) Training vectors, where `n_samples` is the number of samples and `n_features` is the number of features. When using GCV, will be cast to float64 if necessary. y : ndarray of shape (n_samples,) Target values. Will be cast to X's dtype if necessary. sample_weight : float or ndarray of shape (n_samples,), default=None Individual weights for each sample. If given a float, every sample will have the same weight. **params : dict, default=None Parameters to be passed to the underlying scorer. .. versionadded:: 1.5 Only available if `enable_metadata_routing=True`, which can be set by using ``sklearn.set_config(enable_metadata_routing=True)``. See :ref:`Metadata Routing User Guide ` for more details. Returns ------- self : object Fitted estimator. rl)rNr)rUrr9r.)r$r>rfrr-rTrr:s rAr.zRidgeClassifierCV.fitV sgH"&!3!3Aq-PW!3!X!X1mQgo1 AvEE]EfEEE rCrr:)rr0r1rFrr!r2r$r3parampopr%rr.rGrHs@rArCrC sCCJ$  -$zz:,77>$$D2**""5)))) ) $)))))))*\555,,,,,65,,,,,rCrC)NrKrNNN)NFr:)TNrKNNN)NrNrKrFNFFFNNTF)hrFrrdabcrr functoolsrrrnumpyr]scipyrr r scipy.sparser_ sklearn.baser baser rrr exceptionsrmetricsrrmodel_selectionr preprocessingrutilsrrrrrrrutils._array_apirrrr r!utils._param_validationr"r#r$r% utils.extmathr&r' utils.fixesr(utils.metadata_routingr)r*r+r,r-utils.sparsefuncsr.utils.validationr/r0r1_baser3r4r5r6_sagr7rJrvrrrrrrrHrrrrrr6rJrerorwryrrrrrr@rCrUrCrAr]s''''''''""""""""**********,,,,,,&&&&&&PPPPPPPPPPPP++++++55555555************TSSSSSSSSSSS66666666++++++322222SSSSSSSSSSVVVVVVVVVVVV   (  FFFF\   55555p,;;;;|        DDDDN%%%OY-E F^(4D888,G HT4i 8 8 8   JXXX   Xh4???Fq$v6667;K'(#&K!{+.#'1>   @@@@54@N     #b4b4b4b4J@   IIIIIIIIIXttttt njtttnWWWWW1WWWteeeee+ZeeeP$DDDDDv}3DDDB 4D     /   T T T T T  T T T nxxxxx;xxxvzzzzz zzzzOOOOO-|OOOOOrC