^MhLdZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z d Z!d Z"d Z#d Z$dS) a>Trust Region Reflective algorithm for least-squares optimization. The algorithm is based on ideas from paper [STIR]_. The main idea is to account for the presence of the bounds by appropriate scaling of the variables (or, equivalently, changing a trust-region shape). Let's introduce a vector v: | ub[i] - x[i], if g[i] < 0 and ub[i] < np.inf v[i] = | x[i] - lb[i], if g[i] > 0 and lb[i] > -np.inf | 1, otherwise where g is the gradient of a cost function and lb, ub are the bounds. Its components are distances to the bounds at which the anti-gradient points (if this distance is finite). Define a scaling matrix D = diag(v**0.5). First-order optimality conditions can be stated as D^2 g(x) = 0. Meaning that components of the gradient should be zero for strictly interior variables, and components must point inside the feasible region for variables on the bound. Now consider this system of equations as a new optimization problem. If the point x is strictly interior (not on the bound), then the left-hand side is differentiable and the Newton step for it satisfies (D^2 H + diag(g) Jv) p = -D^2 g where H is the Hessian matrix (or its J^T J approximation in least squares), Jv is the Jacobian matrix of v with components -1, 1 or 0, such that all elements of matrix C = diag(g) Jv are non-negative. Introduce the change of the variables x = D x_h (_h would be "hat" in LaTeX). In the new variables, we have a Newton step satisfying B_h p_h = -g_h, where B_h = D H D + C, g_h = D g. In least squares B_h = J_h^T J_h, where J_h = J D. Note that J_h and g_h are proper Jacobian and gradient with respect to "hat" variables. To guarantee global convergence we formulate a trust-region problem based on the Newton step in the new variables: 0.5 * p_h^T B_h p + g_h^T p_h -> min, ||p_h|| <= Delta In the original space B = H + D^{-1} C D^{-1}, and the equivalent trust-region problem is 0.5 * p^T B p + g^T p -> min, ||D^{-1} p|| <= Delta Here, the meaning of the matrix D becomes more clear: it alters the shape of a trust-region, such that large steps towards the bounds are not allowed. In the implementation, the trust-region problem is solved in "hat" space, but handling of the bounds is done in the original space (see below and read the code). The introduction of the matrix D doesn't allow to ignore bounds, the algorithm must keep iterates strictly feasible (to satisfy aforementioned differentiability), the parameter theta controls step back from the boundary (see the code for details). The algorithm does another important trick. If the trust-region solution doesn't fit into the bounds, then a reflected (from a firstly encountered bound) search direction is considered. For motivation and analysis refer to [STIR]_ paper (and other papers of the authors). In practice, it doesn't need a lot of justifications, the algorithm simply chooses the best step among three: a constrained trust-region step, a reflected step and a constrained Cauchy step (a minimizer along -g_h in "hat" space, or -D^2 g in the original space). Another feature is that a trust-region radius control strategy is modified to account for appearance of the diagonal C matrix (called diag_h in the code). Note that all described peculiarities are completely gone as we consider problems without bounds (the algorithm becomes a standard trust-region type algorithm very similar to ones implemented in MINPACK). The implementation supports two methods of solving the trust-region problem. The first, called 'exact', applies SVD on Jacobian and then solves the problem very accurately using the algorithm described in [JJMore]_. It is not applicable to large problem. The second, called 'lsmr', uses the 2-D subspace approach (sometimes called "indefinite dogleg"), where the problem is solved in a subspace spanned by the gradient and the approximate Gauss-Newton step found by ``scipy.sparse.linalg.lsmr``. A 2-D trust-region problem is reformulated as a 4th order algebraic equation and solved very accurately by ``numpy.roots``. The subspace approach allows to solve very large problems (up to couple of millions of residuals on a regular PC), provided the Jacobian matrix is sufficiently sparse. References ---------- .. [STIR] Branch, M.A., T.F. Coleman, and Y. Li, "A Subspace, Interior, and Conjugate Gradient Method for Large-Scale Bound-Constrained Minimization Problems," SIAM Journal on Scientific Computing, Vol. 21, Number 1, pp 1-23, 1999. .. [JJMore] More, J. J., "The Levenberg-Marquardt Algorithm: Implementation and Theory," Numerical Analysis, ed. G. A. Watson, Lecture N)norm)svdqr)lsmr)OptimizeResult)step_size_to_boundfind_active_constraints in_boundsmake_strictly_feasibleintersect_trust_regionsolve_lsq_trust_regionsolve_trust_region_2dminimize_quadratic_1dbuild_quadratic_1devaluate_quadraticright_multiplied_operatorregularized_lsq_operatorCL_scaling_vector compute_gradcompute_jac_scalecheck_terminationupdate_tr_radiusscale_for_robust_loss_functionprint_header_nonlinearprint_iteration_nonlinearctj|tj kr>tj|tjkrt|||||||| | | | | ||St |||||||||| | | | | ||S)N)npallinf trf_no_bounds trf_bounds)funjacx0f0J0lbubftolxtolgtolmax_nfevx_scale loss_function tr_solver tr_optionsverboses W/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/scipy/optimize/_lsq/trf.pytrfr4ps  vbRVGm;bf !5!5; b"b$dHg 9j';; ; b"b"b$dHg 9j';; ;c t||z|| rt||||} ||| fSt|||| \} } tj|}|| t xxdzcc<||z}|| z}|| z}||z}t|||\}}t|||| \}}t||}|dkrd| z | z|z }||kr| |z}n|}nd}d}||kr?t|||||\}}}t|||||\}}||z}||z }||z}n tj }|| z}|| z}t||||} | }||z}|t|z }t|||| \}}||kr| |z}n|}t||||\}}t||d|\}}||z}||z}| |kr | |kr||| fS|| kr ||kr||| fS||| fS)zDSelect the best step according to Trust Region Reflective algorithm.diagrr)s0r8)c) r rr rcopyastypeboolr minrrr r)xJ_hdiag_hg_hpp_hdDeltar(r)thetap_valuep_stridehitsr_hr x_on_bound_to_trto_boundr_stride r_stride_l r_stride_uabr;r_valueag_hag ag_strideag_values r3 select_stepr\sQB $S#s@@@#x'1b"55NHd '#,,C Db  CAMA8OCQJ&c366HAu$ZB;;KHa 8U##H!||%i8+h6 x  )JJJJ  Z$S#ssHHH1a1 q*jA///' x s  !G&JA5LC c3V<<d}*|*dkro|| krh| dkrt=|||2|0|1| |&\}?}&}@n/| dkr)t?|<|=| \}A}:|9|A}?|+|?z}BtA||.|,|-|B|?|+| |||> \}C}D}EtC||Cz||d}F||F}G|dz }t|D}Htj"tj#|Gsd|Hz} | | |Gd }Indtj|G|Gz}I||Iz }*tI| |*|E|H|Hd| zk\}J}Kt|C})tK|*||)t||K||}'|'n|&| |Jz z}&|J} |*dkr|| kh|*dkrr|F}|G}|}|I}|||}|dz }| | |}t |||\}}t ||}|rt||\}}nd})d}*|(dz }(8|'d}'tM||||}LtO||||||!|L|||' S)Nr?rr$?ordexactr regularizeTdF full_matricesr7economicmodegףp= ?r9 initial_alpha)rstep? cost_onlyffffff?)rtol r@costr#r$grad optimality active_masknfevnjevstatus)(r<shapersumrdotr isinstancestrrrrr zerosemptypopsizerrr8rTrrrrrvstackrmaxrrr\r risfiniterrr r)Mr#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r@ff_trueryJrzmnrhorug jac_scalescale scale_invvdvrGg_norm f_augmented J_augmentedreg_termrdalphatermination_status iteration step_normactual_reductionrFrBrCrAUsVufrUrVrPr[lsmr_opgn_hSrOJSB_Sg_SrHrEn_iterp_SrDstepstep_hpredicted_reductionx_newf_new step_h_normcost_new Delta_newratiorxsM r3r"r"sx  A A VVXXF D A D 7DAq mARVCF^^#-aC8811RVAq\\!QA7C((=W-=I0,Q//yy"AKy aB + +EArbAgJJJ)B!G$$JJJ i!S&( ) )E zz !a%RV $ $ $F(1q5//KGhAqz** f  ^^L$77 7S= EII!||   A!!QB//2a!e((( D==!"  a<< %it=M&/ 9 9 9  )TX-=-=  "' ia((  sFUNR% !e BQB   !eKObqb/C gfck22KO+U;;;GAq!A%%BB & +Aq11C 0)#sSDvFFF1S )0Aq%@@C$9uax/.sVh5F4LMMG;; ;;A>D 3+&&(Aaj)))DAqB&r""RVAC&L!%<%<|dz }t|8}?tjtj|>sd|?z}| | |>d }@ndtj|>|>z}@||@z }$t;||$|;|?|?d|zk\}A}Bt|<}#t=|$||#t||B||}!|!n| ||Az z} |A}|$dkr||kL|$dkrr|=}|>}|}|@}|||}|dz }| | |}t |||\}}t ||}|rt||\}}nd}#d}$|"dz }"h|!d}!tj|}CtA||||||%|C|||! S)Nrr^rr$r_rdamprcrdTrerfr`rbFrgrirjr9rlrorprrrt)!r<r|rr}rr~rrrrrrrrr rrrrrrrrrrrrrrrr zeros_liker)Dr#r$r%r&r'r*r+r,r-r.r/r0r1r2r@rrryrrzrrrrurrrrrGrrrdrrrrrrrFrCrArrrrrUrVrPr[ damp_fullrrrOrrrrrrrrrrrrrrrxsD r3r!r!s  A A VVXXF D A D 7DAq mARVCF^^#-aC8811RVAq\\!QA7C((=W-=I0,Q//yy"AKy i E zzF~~fc**^^L$77 7S= EII!||   eaRV$$$ D==!"  a<< %it=M&/ 9 9 9  )TX-=-=  !e   a%C#U333GAq!ABB & +Aq11C 0)#sSD991S )0Aq%@@C$9uax/q8+c1IQ==Y=*==a@D 3+&&(Aaj)))DAqB&r""C#''#,,C!##xG##(>q"aE)@)@)@%vvf$$.sC??Qs#5c3#G#G"G v:DHECJJE AIDv,,K6"+e,,-- {*((=$???u!5!55#h /')<[4%<7 9 9 IuT I!2 $ 477E4"O"O !- UY& &EEM!##xP a  AAVVXXFDAq A AID(#mA&&5aC@@1Q""A C#4Q #B#B yI Q KeN!-""K  $Ffd! # # ##r5)%__doc__numpyr numpy.linalgr scipy.linalgrrscipy.sparse.linalgrscipy.optimizercommonr r r r r rrrrrrrrrrrrrrrr4r\r"r!r5r3rs^^~ $$$$$$)))))) ; ; ; J#J#J#ZA#A#A#H_#_#_#_#_#r5