0PhdZddlmZmZddlZddlmZddl m Z ddl m Z m Z ddlmZmZmZmZddlmZmZmZdd lmZmZdd lmZmZdd lmZmZm Z d d l!m"Z#d dl!m$Z$d dl!m%Z&ej'e(j)Z*dZ+GddeeeZ,dS)a6 ============================================================= Online Latent Dirichlet Allocation with variational inference ============================================================= This implementation is modified from Matthew D. Hoffman's onlineldavb code Link: https://github.com/blei-lab/onlineldavb )IntegralRealN)effective_n_jobs)gammaln logsumexp) BaseEstimatorClassNamePrefixFeaturesOutMixinTransformerMixin _fit_context)check_random_state gen_batchesgen_even_slices)Interval StrOptions)Paralleldelayed)check_is_fittedcheck_non_negative validate_data)_dirichlet_expectation_1d)_dirichlet_expectation_2d) mean_changec ,tj|}|j\}} |jd} |r4|dd|| f|jd} nt j|| f|j} t jt| } |r t j |j|jnd} |r|j }|j }|j }|jt jkrdnd }t|}t |}t j|jj}t'|D]U}|r3|||||d z}|||||d z}n.t j||ddfd}|||f}| |ddf}| |ddf}|dd|f}t'd|D][}|}t j|||z}|t j||z |jz}||||||||krn\|| |ddf<|rBt j|||z}| dd|fxxt j|||z z cc<W| | fS) aE-step: update document-topic distribution. Parameters ---------- X : {array-like, sparse matrix} of shape (n_samples, n_features) Document word matrix. exp_topic_word_distr : ndarray of shape (n_topics, n_features) Exponential value of expectation of log topic word distribution. In the literature, this is `exp(E[log(beta)])`. doc_topic_prior : float Prior of document topic distribution `theta`. max_doc_update_iter : int Max number of iterations for updating document topic distribution in the E-step. mean_change_tol : float Stopping tolerance for updating document topic distribution in E-step. cal_sstats : bool Parameter that indicate to calculate sufficient statistics or not. Set `cal_sstats` to `True` when we need to run M-step. random_state : RandomState instance or None Parameter that indicate how to initialize document topic distribution. Set `random_state` to None will initialize document topic distribution to a constant number. Returns ------- (doc_topic_distr, suff_stats) : `doc_topic_distr` is unnormalized topic distribution for each document. In the literature, this is `gamma`. we can calculate `E[log(theta)]` from it. `suff_stats` is expected sufficient statistics for the M-step. When `cal_sstats == False`, this will be None. rY@g{Gz?FcopydtypeNfloatdoubler)spissparseshapegammaastyper nponesexprzerosdataindicesindptrfloat32cy_mean_changecy_dirichlet_expectation_1dfinfoepsrangenonzerordotTouter)Xexp_topic_word_distrdoc_topic_priormax_doc_update_itermean_change_tol cal_sstats random_state is_sparse_x n_samples n_featuresn_topicsdoc_topic_distr exp_doc_topic suff_statsX_data X_indicesX_indptrctyperdirichlet_expectation_1dr3idx_didscnts doc_topic_dexp_doc_topic_dexp_topic_word_d_last_dnorm_phis Z/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/sklearn/decomposition/_lda.py_update_doc_distributionrV,sb+a..KGIz#)!,HH&,,UD9h:OPPWW G%X  '9h"7qwGGGF4_EEFFM@JS%+17;;;;tI 8 w"*,,GG(E 'K:5A (17   Cy!!!M!M  !HUOhuqy.AABC(5/HUQY,??@DD*Quaaax[))!,CUCZ=D%eQQQh/ 'qqq16688/37q-..  A Fvo/?@@3FH)BF4(?DTDV,W,WWK $ $[/? S S S{6;///AAB$/qqq!  Mvo/?@@3FH qqq#v   "(?D8O"L"L L    Z ((ceZdZUdZideedddgddeeddd gd deeddd gd ed d hgdeeddd gdeedddgdeedddgdeedddgdeedddgdeedddgdeedddgdeedddgdeedddgddegddgddgZe e d< d;ddd dddd d!d"d#d$d%dddd&d'Z e j fd(Zdd9Zed:ZxZS)?LatentDirichletAllocationaTLatent Dirichlet Allocation with online variational Bayes algorithm. The implementation is based on [1]_ and [2]_. .. versionadded:: 0.17 Read more in the :ref:`User Guide `. Parameters ---------- n_components : int, default=10 Number of topics. .. versionchanged:: 0.19 ``n_topics`` was renamed to ``n_components`` doc_topic_prior : float, default=None Prior of document topic distribution `theta`. If the value is None, defaults to `1 / n_components`. In [1]_, this is called `alpha`. topic_word_prior : float, default=None Prior of topic word distribution `beta`. If the value is None, defaults to `1 / n_components`. In [1]_, this is called `eta`. learning_method : {'batch', 'online'}, default='batch' Method used to update `_component`. Only used in :meth:`fit` method. In general, if the data size is large, the online update will be much faster than the batch update. Valid options: - 'batch': Batch variational Bayes method. Use all training data in each EM update. Old `components_` will be overwritten in each iteration. - 'online': Online variational Bayes method. In each EM update, use mini-batch of training data to update the ``components_`` variable incrementally. The learning rate is controlled by the ``learning_decay`` and the ``learning_offset`` parameters. .. versionchanged:: 0.20 The default learning method is now ``"batch"``. learning_decay : float, default=0.7 It is a parameter that control learning rate in the online learning method. The value should be set between (0.5, 1.0] to guarantee asymptotic convergence. When the value is 0.0 and batch_size is ``n_samples``, the update method is same as batch learning. In the literature, this is called kappa. learning_offset : float, default=10.0 A (positive) parameter that downweights early iterations in online learning. It should be greater than 1.0. In the literature, this is called tau_0. max_iter : int, default=10 The maximum number of passes over the training data (aka epochs). It only impacts the behavior in the :meth:`fit` method, and not the :meth:`partial_fit` method. batch_size : int, default=128 Number of documents to use in each EM iteration. Only used in online learning. evaluate_every : int, default=-1 How often to evaluate perplexity. Only used in `fit` method. set it to 0 or negative number to not evaluate perplexity in training at all. Evaluating perplexity can help you check convergence in training process, but it will also increase total training time. Evaluating perplexity in every iteration might increase training time up to two-fold. total_samples : int, default=1e6 Total number of documents. Only used in the :meth:`partial_fit` method. perp_tol : float, default=1e-1 Perplexity tolerance. Only used when ``evaluate_every`` is greater than 0. mean_change_tol : float, default=1e-3 Stopping tolerance for updating document topic distribution in E-step. max_doc_update_iter : int, default=100 Max number of iterations for updating document topic distribution in the E-step. n_jobs : int, default=None The number of jobs to use in the E-step. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. verbose : int, default=0 Verbosity level. random_state : int, RandomState instance or None, default=None Pass an int for reproducible results across multiple function calls. See :term:`Glossary `. Attributes ---------- components_ : ndarray of shape (n_components, n_features) Variational parameters for topic word distribution. Since the complete conditional for topic word distribution is a Dirichlet, ``components_[i, j]`` can be viewed as pseudocount that represents the number of times word `j` was assigned to topic `i`. It can also be viewed as distribution over the words for each topic after normalization: ``model.components_ / model.components_.sum(axis=1)[:, np.newaxis]``. exp_dirichlet_component_ : ndarray of shape (n_components, n_features) Exponential value of expectation of log topic word distribution. In the literature, this is `exp(E[log(beta)])`. n_batch_iter_ : int Number of iterations of the EM step. n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 n_iter_ : int Number of passes over the dataset. bound_ : float Final perplexity score on training set. doc_topic_prior_ : float Prior of document topic distribution `theta`. If the value is None, it is `1 / n_components`. random_state_ : RandomState instance RandomState instance that is generated either from a seed, the random number generator or by `np.random`. topic_word_prior_ : float Prior of topic word distribution `beta`. If the value is None, it is `1 / n_components`. See Also -------- sklearn.discriminant_analysis.LinearDiscriminantAnalysis: A classifier with a linear decision boundary, generated by fitting class conditional densities to the data and using Bayes' rule. References ---------- .. [1] "Online Learning for Latent Dirichlet Allocation", Matthew D. Hoffman, David M. Blei, Francis Bach, 2010 https://github.com/blei-lab/onlineldavb .. [2] "Stochastic Variational Inference", Matthew D. Hoffman, David M. Blei, Chong Wang, John Paisley, 2013 Examples -------- >>> from sklearn.decomposition import LatentDirichletAllocation >>> from sklearn.datasets import make_multilabel_classification >>> # This produces a feature matrix of token counts, similar to what >>> # CountVectorizer would produce on text. >>> X, _ = make_multilabel_classification(random_state=0) >>> lda = LatentDirichletAllocation(n_components=5, ... random_state=0) >>> lda.fit(X) LatentDirichletAllocation(...) >>> # get topics for some given samples: >>> lda.transform(X[-2:]) array([[0.00360392, 0.25499205, 0.0036211 , 0.64236448, 0.09541846], [0.15297572, 0.00362644, 0.44412786, 0.39568399, 0.003586 ]]) n_componentsrNneither)closedr;rbothtopic_word_priorlearning_methodbatchonlinelearning_decaylearning_offset?leftmax_iter batch_sizeevaluate_every total_samplesperp_tolr=r<n_jobsverboser?_parameter_constraints gffffff?g$@g.Ag?gMbP?d)r;r^r_rbrcrfrgrhrirjr=r<rkrlr?c||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ | |_ ||_ ||_||_dSN)rZr;r^r_rbrcrfrgrhrirjr=r<rkrlr?)selfrZr;r^r_rbrcrfrgrhrirjr=r<rkrlr?s rU__init__z"LatentDirichletAllocation.__init__ks(). 0.,.  $,*  .#6   (rWct|j|_d|_d|_|jd|jz |_n |j|_|jd|jz |_ n |j|_ d}d|z }|j |||j|f |d|_ tjt|j |_dS)zInitialize latent variables.rrNrdrFr)r r? random_state_ n_batch_iter_n_iter_r;rZdoc_topic_prior_r^topic_word_prior_r&r' components_r(r*rexp_dirichlet_component_)rtrBr init_gammainit_vars rU_init_latent_varsz+LatentDirichletAllocation._init_latent_varss00ABB   '$'$*;$;D ! !$($8D !  (%(4+<%r?rts rU z4LatentDirichletAllocation._e_step..s{    .G, - -)QQQ,-%($         rWr)rwrrkrmaxrlrr%zipr(vstackr+r|r r}) rtr9r> random_initparallelrkresults doc_topics sstats_listrDrFsstatsr?s ``` @rU_e_stepz!LatentDirichletAllocation._e_steps/>.9Bt))d "$+..  vs1dlQ>N7O7OPPPH(        -QWQZ@@      #&w- K)J//  $"2"8@P@VWWWJ% % %f$ $7 7JJJ,,rWc||dd|\}}|r|j|z|_nvtj|j|jz|j }t||j dz }|xjd|z zc_|xj||j||zzzz c_tj t|j|_ |xjdz c_dS)aEM update for 1 iteration. update `_component` by batch VB or online VB. Parameters ---------- X : {array-like, sparse matrix} of shape (n_samples, n_features) Document word matrix. total_samples : int Total number of documents. It is only used when batch_update is `False`. batch_update : bool Parameter that controls updating method. `True` for batch learning, `False` for online learning. parallel : joblib.Parallel, default=None Pre-initialized instance of joblib.Parallel Returns ------- doc_topic_distr : ndarray of shape (n_samples, n_components) Unnormalized document topic distribution. Tr>rrrrN) rr{r|r(powerrcrxrbr!r%r*rr}) rtr9ri batch_updaterrRrFweight doc_ratios rU_em_stepz"LatentDirichletAllocation._em_steps 8 $D8%  :  #5 BD  X$t'99D  " ":QW " = = = )/2 2 2>t/5a89:  "$+.. VSDL14D-E-E F F F ((J??    ilO"&"4!&%                  sr)r)rtr9rDrRs rU_unnormalized_transformz1LatentDirichletAllocation._unnormalized_transforms&"\\!5\QQrW normalizect|||dd}||}|r-||dddtjfz}|S)aTransform data X according to the fitted model. .. versionchanged:: 0.18 `doc_topic_distr` is now normalized. Parameters ---------- X : {array-like, sparse matrix} of shape (n_samples, n_features) Document word matrix. normalize : bool, default=True Whether to normalize the document topic distribution. Returns ------- doc_topic_distr : ndarray of shape (n_samples, n_components) Document topic distribution for X. Fz#LatentDirichletAllocation.transformrraxisN)rrrsumr(newaxis)rtr9rrDs rU transformz#LatentDirichletAllocation.transforms&   % % ,Q &  66q99  J 222::111bj=I IOrWcX|||||S)a Fit to data, then transform it. Fits transformer to `X` and `y` and returns a transformed version of `X`. Parameters ---------- X : array-like of shape (n_samples, n_features) Input samples. y : array-like of shape (n_samples,) or (n_samples, n_outputs), default=None Target values (None for unsupervised transformations). normalize : bool, default=True Whether to normalize the document topic distribution in `transform`. Returns ------- X_new : ndarray array of shape (n_samples, n_features_new) Transformed array. r)rr)rtr9rrs rU fit_transformz'LatentDirichletAllocation.fit_transforms).xx1~~''Y'???rWcd}tj|}|j\}}|jjd}d} t |} t |j} |j} |j} |r|j}|j}|j }td|D]}|r3|||||dz}|||||dz}n.tj ||ddfd}|||f}| |ddtj f| dd|fz}t|d}| tj||z } | || || |jz } |rt#|j|z }| |z} | || |j| |z } | S)a Estimate the variational bound. Estimate the variational bound over "all documents" using only the documents passed in as X. Since log-likelihood of each word cannot be computed directly, we use this bound to estimate it. Parameters ---------- X : {array-like, sparse matrix} of shape (n_samples, n_features) Document word matrix. doc_topic_distr : ndarray of shape (n_samples, n_components) Document topic distribution. In the literature, this is called gamma. sub_sampling : bool, default=False Compensate for subsampling of documents. It is used in calculate bound in online learning. Returns ------- score : float c 6tj||z |z}|tjt|t|z z }|tjt||zttj|dz z }|S)Nr)r(rr)priordistrdirichlet_distrsizescores rU_loglikelihoodz?LatentDirichletAllocation._approx_bound.._loglikelihood&s}FEEM_<==E RVGENNWU^^;<< >>? $ HHT3VDDDE $ XXh4???@$ xx!T)DDDE$ 88HdDKKKL$ ((4DCCCD$ XXdAtF;;;<$ HHT1d6BBBC$ 1d6 J J JK$ 4"$ I;$ (!$D*#)%#)#)#)#)#)J35*    :@-@-@-@-D4444l.\555---65-^\555PPP65Pd")-8@D@@@@@2MMM^2*,*,*,*,XLLLL8))X)))))rWrY)-rnumbersrrnumpyr( scipy.sparserr#joblibr scipy.specialrrbaser r r r utilsr rrutils._param_validationrrutils.parallelrrutils.validationrrr_online_lda_fastrr1rrr0r2r!r3EPSrVrYrWrUrs#"""""""######,,,,,,,, EDDDDDDDDD::::::::........QQQQQQQQQQbhuoou)u)u)p[ )[ )[ )[ )[ )#%5}[ )[ )[ )[ )[ )rW