M/PhAZdZddlmZddlZddlmZddZdZdd Z dd Z dd Z ddZ dS)aextra statistical function and helper functions contains: * goodness-of-fit tests - powerdiscrepancy - gof_chisquare_discrete - gof_binning_discrete Author: Josef Perktold License : BSD-3 changes ------- 2013-02-25 : add chisquare_power, effectsize and "value" )lrangeN)statsctj|}tj|}t|ts|}n<|dkrd}n3|dkrd}n*|dkrd}n!|dkrd}n|d krd }nt d tj|| }|} |jdkrMtj|}|dkr|j} |j dkr!tj|}|dkr|j}tj tj|| |d dr |d| zz } nCtj tj|| dd dr|} | |z}nt d|j |} |j || krt d|dkr;d|ztj|d| zz tj ||z z| z} nx|dkr;d|ztj|d| zz tj ||z z| z} n7d|z|z |dzz tj|d| zz ||z |zdz z| z} | tj| | dz |z fS)aRCalculates power discrepancy, a class of goodness-of-fit tests as a measure of discrepancy between observed and expected data. This contains several goodness-of-fit tests as special cases, see the description of lambd, the exponent of the power discrepancy. The pvalue is based on the asymptotic chi-square distribution of the test statistic. freeman_tukey: D(x|\theta) = \sum_j (\sqrt{x_j} - \sqrt{e_j})^2 Parameters ---------- o : Iterable Observed values e : Iterable Expected values lambd : {float, str} * float : exponent `a` for power discrepancy * 'loglikeratio': a = 0 * 'freeman_tukey': a = -0.5 * 'pearson': a = 1 (standard chisquare test statistic) * 'modified_loglikeratio': a = -1 * 'cressie_read': a = 2/3 * 'neyman' : a = -2 (Neyman-modified chisquare, reference from a book?) axis : int axis for observations of one series ddof : int degrees of freedom correction, Returns ------- D_obs : Discrepancy of observed values pvalue : pvalue References ---------- Cressie, Noel and Timothy R. C. Read, Multinomial Goodness-of-Fit Tests, Journal of the Royal Statistical Society. Series B (Methodological), Vol. 46, No. 3 (1984), pp. 440-464 Campbell B. Read: Freeman-Tukey chi-squared goodness-of-fit statistics, Statistics & Probability Letters 18 (1993) 271-278 Nobuhiro Taneichi, Yuri Sekiya, Akio Suzukawa, Asymptotic Approximations for the Distributions of the Multinomial Goodness-of-Fit Statistics under Local Alternatives, Journal of Multivariate Analysis 81, 335?359 (2002) Steele, M. 1,2, C. Hurst 3 and J. Chaseling, Simulated Power of Discrete Goodness-of-Fit Tests for Likert Type Data Examples -------- >>> observed = np.array([ 2., 4., 2., 1., 1.]) >>> expected = np.array([ 0.2, 0.2, 0.2, 0.2, 0.2]) for checking correct dimension with multiple series >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected, lambd='freeman_tukey',axis=1) (array([[ 2.745166, 2.745166]]), array([[ 0.6013346, 0.6013346]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected,axis=1) (array([[ 2.77258872, 2.77258872]]), array([[ 0.59657359, 0.59657359]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected, lambd=0,axis=1) (array([[ 2.77258872, 2.77258872]]), array([[ 0.59657359, 0.59657359]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected, lambd=1,axis=1) (array([[ 3., 3.]]), array([[ 0.5578254, 0.5578254]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, 10*expected, lambd=2/3.0,axis=1) (array([[ 2.89714546, 2.89714546]]), array([[ 0.57518277, 0.57518277]])) >>> powerdiscrepancy(np.column_stack((observed,observed)).T, expected, lambd=2/3.0,axis=1) (array([[ 2.89714546, 2.89714546]]), array([[ 0.57518277, 0.57518277]])) >>> powerdiscrepancy(np.column_stack((observed,observed)), expected, lambd=2/3.0, axis=0) (array([[ 2.89714546, 2.89714546]]), array([[ 0.57518277, 0.57518277]])) each random variable can have different total count/sum >>> powerdiscrepancy(np.column_stack((observed,2*observed)), expected, lambd=2/3.0, axis=0) (array([[ 2.89714546, 5.79429093]]), array([[ 0.57518277, 0.21504648]])) >>> powerdiscrepancy(np.column_stack((observed,2*observed)), expected, lambd=2/3.0, axis=0) (array([[ 2.89714546, 5.79429093]]), array([[ 0.57518277, 0.21504648]])) >>> powerdiscrepancy(np.column_stack((2*observed,2*observed)), expected, lambd=2/3.0, axis=0) (array([[ 5.79429093, 5.79429093]]), array([[ 0.21504648, 0.21504648]])) >>> powerdiscrepancy(np.column_stack((2*observed,2*observed)), 20*expected, lambd=2/3.0, axis=0) (array([[ 5.79429093, 5.79429093]]), array([[ 0.21504648, 0.21504648]])) >>> powerdiscrepancy(np.column_stack((observed,2*observed)), np.column_stack((10*expected,20*expected)), lambd=2/3.0, axis=0) (array([[ 2.89714546, 5.79429093]]), array([[ 0.57518277, 0.21504648]])) >>> powerdiscrepancy(np.column_stack((observed,2*observed)), np.column_stack((10*expected,20*expected)), lambd=-1, axis=0) (array([[ 2.77258872, 5.54517744]]), array([[ 0.59657359, 0.2357868 ]])) loglikeratior freeman_tukeygpearsonmodified_loglikeratio cressie_readgUUUUUU?znlambd has to be a number or one of loglikeratio, freeman_tukey, pearson, modified_loglikeratio or cressie_read)axis:0yE>)rtolatol?zZobserved and expected need to have the same number of observations, or e needs to add to 1z:observed and expected need to have the same number of bins)nparray isinstancestr ValueErrorsumsize atleast_2dTndimallcloseshapelogrchi2sf) observedexpectedlambdrddofoeanntpkD_obss U/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/stats/gof.pypowerdiscrepancyr0sr A A eS ! !F  N " "AA o % %AA i  AA - - -AA n $ $AAEFF F qtA Bvaxx M!   199B 6Q;; a  AqyyC {26!$'''A>>>K s2vJ RVAD)))14a @ @ @K  FJKK K  Awt}*++ + Avv!bfQBZ"&1++5DAAAA b!bfQBZ"&1++5DAAAA!Aqs bfQBZAaC!8a<%@tLLLL %*--ac$h// //c zt|}d}d|z }tt|jdt |jddz}d} t|jdg} g} |D]V} |j| g|R} | | z |dz kr:| | | | | z | } | d|z krnW| d|jkr2| |j| d| z tj | } tj | } | d z}|j|d<tj ||\}}|j| g|R}tj tj ||| z\}}||||kd |d t|d t|fS) aperform chisquare test for random sample of a discrete distribution Parameters ---------- distname : str name of distribution function arg : sequence parameters of distribution alpha : float significance level, threshold for p-value Returns ------- result : bool 0 if test passes, 1 if test fails Notes ----- originally written for scipy.stats test suite, still needs to be checked for standalone usage, insufficient input checking may not run yet (after copy/paste) refactor: maybe a class, check returns, or separate binning from test results rr r+=r rzchisquare - test for z at arg = z with pval = )lenrmaxr)minbcdfappendrr histogramr chisquarer)distfnargrvsalphamsgr*nsuppwsupp distsupportlastdistsuppdistmassiicurrenthistsuppfreqhsuppcdfschispvals r/gof_chisquare_discreterRs: CA E IEVXu--s68T/B/BQ/FGGK DFHe$$%HH&*R%%%% T>U5[ ( ( OOB    OOGdN + + +D!E'""|vx!!!$x!!Hx!!H}H(HQK,s8,,KD% 6:h $ $ $ $D/"(4..8<U5[ ( ( OOB    OOGdN + + +D!E'""|vx!!!$x!!Hx!!H}H(HQKc(++JD 6:h $ $ $ $D 8D>>1X:x //r1Tc,tj|}t|}|d}|?tj|t }||t |z tj|t }||z dz|z d}|dkr'tj ||dz |z }n-tj ||dz |z |dz|z}|r||fS||fS)achisquare goodness-of-fit test The null hypothesis is that the distance between the expected distribution and the observed frequencies is ``value``. The alternative hypothesis is that the distance is larger than ``value``. ``value`` is normalized in terms of effect size. The standard chisquare test has the null hypothesis that ``value=0``, that is the distributions are the same. Notes ----- The case with value greater than zero is similar to an equivalence test, that the exact null hypothesis is replaced by an approximate hypothesis. However, TOST "reverses" null and alternative hypothesis, while here the alternative hypothesis is that the distance (divergence) is larger than a threshold. References ---------- McLaren, ... Drost,... See Also -------- powerdiscrepancy scipy.stats.chisquare rNrr ) rasarrayr7remptyfloatfillrr!r"ncx2) f_obsf_expvaluer& return_basicn_binsnobschisqpvalues r/r>r>Ts@ Ju  E ZZF 99Q<>%!d!2 3 3D JMM$ T 1;>D3H I IE Lr1ctj|t}tj|t}|||z }|||z }||z dz|z |}|D|\}}||z |z |}tj||z|z |z |dz z d}|rtj|S|S)aseffect size for a chisquare goodness-of-fit test Parameters ---------- probs0 : array_like probabilities or cell frequencies under the Null hypothesis probs1 : array_like probabilities or cell frequencies under the Alternative hypothesis probs0 and probs1 need to have the same length in the ``axis`` dimension. and broadcast in the other dimensions Both probs0 and probs1 are normalized to add to one (in the ``axis`` dimension). correction : None or tuple If None, then the effect size is the chisquare statistic divide by the number of observations. If the correction is a tuple (nobs, df), then the effectsize is corrected to have less bias and a smaller variance. However, the correction can make the effectsize negative. In that case, the effectsize is set to zero. Pederson and Johnson (1990) as referenced in McLaren et all. (1994) cohen : bool If True, then the square root is returned as in the definition of the effect size by Cohen (1977), If False, then the original effect size is returned. axis : int If the probability arrays broadcast to more than 1 dimension, then this is the axis over which the sums are taken. Returns ------- effectsize : float effect size of chisquare test rNrr)rrVrXrmaximumsqrt) probs0probs1 correctioncohenrd2r`dfdiffs r/chisquare_effectsizertsFZ & &F Z & &F fjj&& &F fjj&& &F F?Q  ' , ,T 2 2Bb&F*//55 ZdT)B.4"9=q A A wr{{ r1)rrr)r3)NrrT)rcr)NTr) __doc__statsmodels.compat.pythonrnumpyrscipyrr0rRrTr>rirtr1r/rzs&-,,,,,N0N0N0N0jDCDCDCNP0P0P0P0h3333l++++\222222r1