M/Ph@ dZddlmZmZddlmZddlZddlm Z ddl m Z dFdZ dZdGd Zd Zd Zd ZdHdZdHdZdIdZdHdZdHdZdHdZdJdZdKdZedkrededgdZgdZeD]ZeeeeD]ZeeeeeeeeegdZ e j!d e j"d!e j#d"ej$ddd#Z%e j&e%ee%e j!d e j"d$e j#d"ej$ddd%Z%e j&e%eeee%de%z ej'gd&gd'gd(gZ(e()dZ*e()dZ+ee*Z,ee+Z-ee(Z.ee*e+e(Z/ee+e*e(Z0edej1ej2e*e+zZ3edej1ej2e+e*zZ4ee*e+e(Z5ed)ee,e-e.e/e0e3e4e5ed*ej'gd+ZeeZ6eed,ed-ed.ed/ed0ed1e*zed2de*dz Z7ed3e7zee7de7z gZ8ed4e8zed5e8e7ej9dzze/fzed6ed7e7d8d9e,dz ej9d:z d8ed;ed<ej'gd=gd>gd?gZ:ee:ed@edAej;dee:dee:dgdz ej9d:z zedBej'gdCgdDgdEgZ!#4#44T9:: ~~q BF1II%u%%^A&& d8D>>Aw"aq + +AAw%%$*a!!Aw! $*a!! OrcTtj|tj|z S)z There is a one-to-one transformation of the entropy value from a log base b to a log base a : H_{b}(X)=log_{b}(a)[H_{a}(X)] Returns ------- log_{b}(a) )r r)rbs r logbasechanger;s 6!99RVAYY rc<ttjd|zS)z$ Converts from nats to bits r;r ers r natstobitsr@s q ! !A %%rc<tdtj|zS)z$ Converts from bits to nats r=r>rs r bitstonatsrBs BD ! !A %%rr=cVtj|}tj|dkrtj|dkstdtjtj|tj|z }|dkrtd||zS|S)aC This is Shannon's entropy Parameters ---------- logbase, int or np.e The base of the log px : 1d or 2d array_like Can be a discrete probability distribution, a 2d joint distribution, or a sequence of probabilities. Returns ----- For log base 2 (bits) given a discrete distribution H(p) = sum(px * log2(1/px) = -sum(pk*log2(px)) = E[log2(1/p(X))] For log base 2 (bits) given a joint distribution H(px,py) = -sum_{k,j}*w_{kj}log2(w_{kj}) Notes ----- shannonentropy(0) is defined as 0 r r&px does not define proper distributionr=)r r r ValueErrorr nan_to_numlog2r;)pxlogbaseentropys rshannonentropyrKs2 BB 6"'??C"&q//CABBBvbmBrwr{{N33444G!||Qw'''11rc*tj|}tj|dkrtj|dkstd|dkr&t d| tj|zStj| S)z Shannon's information Parameters ---------- px : float or array_like `px` is a discrete probability distribution Returns ------- For logbase = 2 np.log2(px) r rrDr=)r r rrEr;rG)rHrIs r shannoninforMs BB 6"'??C"&q//CABBB!||q)))BGBKK77}rc |t|rt|std|t|std|tj||}tj|tjtj||z z}|dkr|Std||zS)a Return the conditional entropy of X given Y. Parameters ---------- px : array_like py : array_like pxpy : array_like, optional If pxpy is None, the distributions are assumed to be independent and conendtropy(px,py) = shannonentropy(px) logbase : int or np.e Returns ------- sum_{kj}log(q_{j}/w_{kj} where q_{j} = Y[j] and w_kj = X[k,j] 1px or py is not a proper probability distributionN&pxpy is not a proper joint distribtionr=)r!rEr outerrrFrGr;)rHpypxpyrIcondents r condentropyrUs(   NM"$5$5NLMMM  d 3 3ABBB |x2fTBM"'"T'*:*:;;;<   NM"$5$5NLMMM  d 3 3ABBB |x2 BD' 2 2 2b"dG444 56rr Rc t|stdt|}|dkr*t|}|dkrt d||zS|Sdt |vs|tjkr'tj tj | S||z}tj | }|dkr dd|z z |zSdd|z z t d|z|zS)as Renyi's generalized entropy Parameters ---------- px : array_like Discrete probability distribution of random variable X. Note that px is assumed to be a proper probability distribution. logbase : int or np.e, optional Default is 2 (bits) alpha : float or inf The order of the entropy. The default is 1, which in the limit is just Shannon's entropy. 2 is Renyi (Collision) entropy. If the string "inf" or numpy.inf is specified the min-entropy is returned. measure : str, optional The type of entropy measure desired. 'R' returns Renyi entropy measure. 'T' returns the Tsallis entropy measure. Returns ------- 1/(1-alpha)*log(sum(px**alpha)) In the limit as alpha -> 1, Shannon's entropy is returned. In the limit as alpha -> inf, min-entropy is returned. z+px is not a proper probability distributionr r=inf) r!rEfloatrKr;strlowerr r`rrr)rHalpharImeasuregenents r renyientropyrgks :   HFGGG %LLE zz## a<< G,,v5 5 #e**""$$ $ $rvbzz"""" UB VBFFHH  F!||!E'{V##!E'{]1g666??rTcdS)a Generalized cross-entropy measures. Parameters ---------- px : array_like Discrete probability distribution of random variable X py : array_like Discrete probability distribution of random variable Y pxpy : 2d array_like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) measure : str, optional The measure is the type of generalized cross-entropy desired. 'T' is the cross-entropy version of the Tsallis measure. 'CR' is Cressie-Read measure. N)rHrRrSrdrIres rgencrossentropyrksr__main__zQFrom Golan (2008) "Information and Entropy Econometrics -- A Review and Synthesisz Table 3.1)皙?rmrmrmrm)S㥛?g;On?g'1Z?gK?gMbp?) gh㈵>g-C6?gMbP?g{Gz?g?g333333?rmg?g333333?gffffff?g?g??o Information ProbabilityiEntropye)rrUUUUUU?)qq?rvrv)gqq?rvUUUUUU?z Table 3.3zdiscretize functions)2g3333335@g@F@g?@g3@gLD@gYC@g333333&@g/@gfffff?@g9@g3333334@gffffff,@g8@g5@g&@g2@L0@g3333336@g333333@g;@rxǧA@-@g1@g333333<@gffffff0@g0@gG@g#@g2@g @@g:@g0@g333333@gffffff5@g4@gL=@ryg @g6@g)@gfffff:@g9@gfffff6@gffffff&@g333334@g333333:@g"@g%@g333333/@z0Example in section 3.6 of Golan, using table 3.3z'Bounding errors using Fano's inequalityz"H(P_{e}) + P_{e}log(K-1) >= H(X|Y)zor, a weaker inequalityzP_{e} >= [H(X|Y) - 1]/log(K)z P(x) = %sz?X = 3 has the highest probability, so this is the estimate Xhatz1The probability of error Pe is 1 - p(X=3) = %0.4gzH(Pe) = %0.4g and K=3z-H(Pe) + Pe*log(K-1) = %0.4g >= H(X|Y) = %0.4gzor using the weaker inequalityzPe = z0.4gz >= [H(X) - 1]/log(K) = z>Consider now, table 3.5, where there is additional informationz.The conditional probabilities of P(X|Y=y) are )r{g?)rururu)rwruroz2The probability of error given this information iszPe = [H(X|Y) -1]/log(K) = %0.4gz+such that more information lowers the error)gV-?gV-?gw/?)g(\?g+?g%C?)gzG?rngPn?)N)r"N)r=)Nr=)r r=r^)r r=rh)=__doc__statsmodels.compat.pythonrrscipyrnumpyr matplotlibrplt scipy.specialrr r!r8r;r@rBrKrMrUrYr[r]rgrk__name__printr Yr7psubplotylabelxlabellinspacexplotarraywrrHrRH_XH_YH_XY H_XgivenY H_YgivenXr?rJD_YXD_XYI_XYdiscXpeH_perGw2mean markovchainrjrrrs  D10000000$$$$$$333333 !!!!H####P   &&& &&&    F,3333@<&&&&P(6(6(6(6\.@.@.@.@f, z E  E+ A"""A  kk!nn  kk!nn E..   E..  ;;;ACKCJ}CJ} AaA CHQ ACKCJyCJ} AaA CHQ^TT!AaC[[11222 ***---.?.?.?@AAA qB qB .  C .  C >!  D Br!$$I Br!$$I =24 r2!6!6 6D =24 r2!6!6 6D :bA  D E+ E#c4ItT4@@@ E !!!bh:::;;A JqMME E E <=== E 3444 E ./// E #$$$ E ())) E+  E KLLL RUB E = BCCC >2ad) $ $D E !D ())) E 9 "WRWQZZ-  + ,--- E *+++ E N" N N NC!GWRWQZZ3G N N NOOO E JKKK E :;;; :::.../?/?/?@ A AB E"III E >??? E +>>"Q%;P;PQ_Q_`bcd`eQfQf8g0h0hij0jlslnlstulvlv/v wxxx E 7888"(,,,-=-=-=>N>N>NOPPKKKir