Q/PhdddlmZddlmZddlmZddlmZddlm Z dZ dZ dZ dd Z dd Zd S))division)tee) itemgetter) defaultdict)logctt|d|zttd|z d||z zzS)Ng|=)rmax)knxs V/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/wordcloud/tokenization.pylrsC s1e}}   !CAE5(9(9$:$:a!e$D DDc||ks||krdS|}|}|}|}||z }||z } ||z ||z z } t|||t||z ||z |zt||| z t||z ||z | z } d| zS)zCollocation scorer)r) count_bigramcount1count2n_wordsNc12c1c2pp1p2scores rrrs&Gv--qA C B B QA rB s(q2v B sB]]QrCxR33 3b"~~ !"s(AFB 7 78E :rcft|\}}t|dt||SN)rnextzip)iterableabs rpairwiser&s. x==DAqDMMM q!99rTcFtfdt|D}tfd|D}t|}t||\}}td|D|\} } |} | D]\} } t | d}||d}||d}t| | || ||}||kr7||xx| | zcc<||xx| | zcc<| | || <t|D]\}} | dkr||=|S)Nc3RK|]!}tfd|D|V"dS)c3DK|]}|vVdSr lower.0w stopwordss r z1unigrams_and_bigrams...+s18[8[TUi9O8[8[8[8[8[8[rN)any)r.rr0s rr1z'unigrams_and_bigrams..+sF\\S8[8[8[8[YZ8[8[8[5[5[\1\\\\\\rc3HK|]}|v|VdSr r+r-s rr1z'unigrams_and_bigrams..,s5CC! (B(BA(B(B(B(BCCr)normalize_pluralsc8g|]}d|S) )join)r.bigrams r z(unigrams_and_bigrams..1s$000f&  000rr6rr ) listr&lenprocess_tokenscopyitemstuplesplitr,r)wordsr0r4collocation_thresholdbigramsunigramsrcounts_unigrams standard_formcounts_bigramsstandard_form_bigrams orig_counts bigram_stringcountr8word1word2collocation_scorewords ` runigrams_and_bigramsrP's\\\\huoo\\\\\GCCCCuCCCCCH(mmG%3$5&7&7&7"O],:00000+------)N)"&&((K!/ 4 4 6 6 K K u}**3//00fQioo//0fQioo//0!%U);[=OQXYY 4 4 4 E " " "n]&C C " " " E " " "n]&C C " " "-;M-JOM *O113344&& e A::% rctt}|D]:}|}||}||ddz||<;|ri}t |D]}|dr{|dsf|dd}||vrX||} ||} | D]+\}} |dd} | | d| z| | <,|||<||=i} i}td}|D]W\}}t||d}t| | |<|||<X|r7|D]"\}} || ||<#| |fS)aNormalize cases and remove plurals. Each word is represented by the most common case. If a word appears with an "s" on the end and without an "s" on the end, the version with "s" is assumed to be a plural and merged with the version without "s" (except if the word ends with "ss"). Parameters ---------- words : iterable of strings Words to count. normalize_plurals : bool, default=True Whether to try and detect plurals and remove trailing "s". Returns ------- counts : dict from string to int Counts for each unique word, with cases represented by the most common case, and plurals removed. standard_forms : dict from string to string For each lower-case word the standard capitalization. rr sssN)key) rdictr,getr:keysendswithr>rr sumvalues)rAr4drO word_lower case_dictmerged_pluralsrU key_singular dict_plural dict_singularrKsingular fused_casesstandard_casesitem1firstplurals rr<r<Ks: DA55ZZ\\ jM #--a0014 $>>  C||C  d);); "3B3x 1$$"#C&K$%lOM'2'8'8':':DD e#'9)--h::UB&h//*6N3'#KN qMME!"++ IIOO%%5111!4 !1!1!3!344 E%*z""F . 4 4 6 6 F F FH%3HNN4D4D%EN6 " "  &&rN)Tr')T) __future__r itertoolsroperatorr collectionsrmathrrrr&rPr<rrros######EEE "!!!!H?'?'?'?'?'?'r