MhYXddlmZddlZddlmZddlmZddlmZm Z m Z m Z ddl m Z mZmZmZddlmZdd lmZmZdd lmZmZmZmZmZmZmZejd Zej Z!e!"ej#d  d.d/d$Z$ d.d0d'Z% d.d1d*Z& d2d3d-Z'dS)4) annotationsN)PathLike)BinaryIO)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bom is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s皙?TF皙? sequencesbytes | bytearraystepsint chunk_size thresholdfloat cp_isolationlist[str] | None cp_exclusionpreemptive_behaviourboolexplainlanguage_thresholdenable_fallbackreturnrc Ft|ttfs/tdt ||rJt j} t tt tt|} | dkrt d|rEt tt | p tjt#t%|dddgdgS|At td d |d |D}ng}|At td d |d |D}ng}| ||zkr't td||| d}| }|dkr| |z |krt+| |z }t|t,k} t|t.k} | r4t td| n5| r3t td| g}|rt1|nd}|6||t td|t5}g}g}d}d}d}t#}t#}t7|\}}|D||t tdt|||dd|vr|d|t8zD]}|r||vr |r||vr||vr||d}||k}|ot=|}|dvr$|s"t td|l|dvr$|s"t td| t?|}n8#t@tBf$r$t td|YwxYw | rS|durOtE|dur|dt+dn#|t|t+d|n,tE|dur|n|t|d|}nx#tFtHf$rd}t|tHs/t td|tE|||Yd}~d}~wwxYwd} |D]}!tK||!rd} n| r$t td||!tM|sdnt|| t+| |z }"|o|duot|| k}#|#r!t td|t+t|"d z }$tO|$d!}$d}%d}&g}'g}( tQ|||"|||||| D]y})|'|)|(tS|)||duodt|cxkod!knc|(d"|kr|%dz }%|%|$ks|r|durnznJ#tF$r=}t td#|tE||$}%d}&Yd}~nd}~wwxYw|&s| r|s |t+d$d*|d%&n\#tF$rO}t td'|tE|||Yd}~#d}~wwxYw|(rtW|(t|(z nd}*|*|ks|%|$kr||t td(||%tY|*d)zd*+| r3|dd|fvr,|&s*t%|||dg||,}+||kr|+}n |dkr|+}n|+}t td-|tY|*d)zd*+|st[|},nt]|},|,rAt td.|tE|,g}-|dkrB|'D]?})t_|)||,rd/|,nd}.|-|.@ta|-}/|/r4t td0|/|t%|||*||/| dus||ddfvr|nd|,}0||0||ddfvr|*d1kr|*dkrmt d2|0j1|r9t tt | t#|0gcS||0t|r|||vrd|vrd|vr|2}1t d2|1j1|r9t tt | t#|1gcS||krnt d3||r9t tt | t#||gcSt|dkr|s|s|r t td4|r6t d5|j1||n{|r||r|r|j3|j3ks|0t d6||n1|r/t d7|||rDt d8|2j1t|dz nt d9|r9t tt | |S):af Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. Custom logging format and handler can be set manually. z3Expected object of type bytes or bytearray, got: {}rzzfrom_bytes..["DDD "e,,DDDzacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c.g|]}t|dSr2r3r4s r7r8zfrom_bytes..fr9r:z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.rz>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_16utf_32z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.>utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.z2Encoding %s does not provide an IncrementalDecodergA)encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sgj@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigits)preemptive_declarationz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {},z We detected language {} using {}rz.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4 isinstance bytearraybytes TypeErrorformattypeloggerlevel addHandlerexplain_handlersetLevelrlendebug removeHandlerloggingWARNINGrrlogjoinr!r r rappendsetrr addrrModuleNotFoundError ImportErrorstrUnicodeDecodeError LookupErrorrrangemaxrrdecodesumroundrr rr r@best fingerprint)2rr r"r#r%r'r(r*r+r,previous_logger_levellengthis_too_small_sequenceis_too_large_sequenceprioritized_encodingsspecified_encodingtestedtested_but_hard_failuretested_but_soft_failurefallback_ascii fallback_u8fallback_specifiedresultsearly_stop_results sig_encoding sig_payload encoding_ianadecoded_payloadbom_or_sig_availablestrip_sig_or_bomis_multi_byte_decoderesimilar_soft_failure_testencoding_soft_failedr_multi_byte_bonusmax_chunk_gave_upearly_stop_countlazy_str_hard_failure md_chunks md_ratioschunkmean_mess_ratiofallback_entrytarget_languages cd_ratioschunk_languagescd_ratios_merged current_matchprobable_results2 r7 from_bytesr!s< i)U!3 4 4  A H HY     %+\/***i..F {{ STTT  F   1 1 1 OO1DW_ E E E|IwUBPRSSTUUU  5 IIl # #    ED|DDD    6 IIl # #    ED|DDD   *u$%%  l         qyyVe^j00%(( "%i..3E"E"%i..4D"D    L S S         W ^ ^     (*.BKy)))t%$$%7888  N    uuF)+)+*.N'+K.2,..G)7)9)9 3I > >L+$$\222  W          )))+++$$W---.?~<~<  M==   M\99  F " "  =!!!&*%1]%B!5" :Q ; ;  0 0 09M 0 JJn     I % %.B % JJd     *@*O*O ! !#[1    JJD    H   $ )>%)G)G,u44"+CII+..&s;'7'7#d))'CD* #&,u44" &s;'7'7'9'9:* ####K0   a--  O!FF  $ * *= 9 9 9 HHHH +0!$;   ],@AA ,0)  %  JJi$      ) ?AAs;/?/?       " .t+ .O$$v-    JJ-    "%SWWq[!1!1 1155 ! %!  ' ),$ %      '''  !4GA\1B1B,G,G,G,Ga,G,G,G,GR=I--$)$$(999(:-=-F-FE  ) ) ) JJsA     1 $( ! ! ! ! ! ! )& % *  #d))++&--mH-MMMM%    t!FF  (..}=== EN!VY#i..!@!@SV i ' '+;?P+P+P # * *= 9 9 9 JJ0 o+Q777      1!gw8J%KKK-L".!#+="""!$666)7&&"g--%3NN"0K   K  /C' 3 3 3    % D*<]*K*K  4]CC    JJ8??!3'7#8#8     G # #" 2 2"1&2BLCHH-...##   11111)<<   JJ299$m   %    *U22$);Wg(NNN  #5   " }%%% 0'7C C C#%%#%% D!*;((999OO$9:::%}o66666  % %m 4 4 4 " # # 5#+/AV/K/K6!!6!!,>,C,C,E,EO LL@(    7$$_555 5666!?"344 4 4 4 L ( ( LL1     7$$_555 5666!7=#9":;; ; ; ; ) 7||q  . ,>  JJa     + LLI"+    NN- . . . .  ++34#4 +~/III' LLU V V V NN; ' ' ' '  + LLU V V V NN> * * *V k LLNN # LL1      TUUU/_----... Ns^ Q2RRBTV (AVV 6B\ ]3]  ],^ _A__fprc Vt|||||||||| S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rread) rr r"r#r%r'r(r*r+r,s r7from_fpr s<       r:pathstr | bytes | PathLikec t|d5} t| ||||||||| cdddS#1swxYwYdS)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openr) rr r"r#r%r'r(r*r+r,rs r7 from_pathr>s dD    R                               s 6::fp_or_path_or_payload!PathLike | str | BinaryIO | bytesc t|ttfrt||||||||||  } nOt|tt frt ||||||||||  } nt||||||||||  } | S)a) Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string. Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match are disabled to be stricter around ASCII-compatible but unlikely to be a string. ) r r"r#r%r'r(r*r+r,)rKrbrrrMrLrr) rr r"r#r%r'r(r*r+r,guessess r7 is_binaryr]s"'#x99,  !!%%!51+            !!%%!51+     !!%%!51+    ;r:) rrrNNTFrT)rrr r!r"r!r#r$r%r&r'r&r(r)r*r)r+r$r,r)r-r)rrr r!r"r!r#r$r%r&r'r&r(r)r*r)r+r$r,r)r-r)rrr r!r"r!r#r$r%r&r'r&r(r)r*r)r+r$r,r)r-r) rrrNNTFrF)rrr r!r"r!r#r$r%r&r'r&r(r)r*r)r+r$r,r)r-r))( __future__rrYosrtypingrcdrrr r constantr r r rmdrmodelsrrutilsrrrrrrr getLoggerrQ StreamHandlerrT setFormatter Formatterrrrrr:r7rsc"""""" RQQQQQQQQQQQ00000000  / 0 0'''))GABB%)%)!% # |||||B%)%)!% # @%)%)!% #      B%)%)!% #!???????r: