M/Phm' dZddlmZddlZdZdZddZGddZe d krddl m Z d Z eje Zd gZd evr}eeeeed eeedejeeZeeeZe jeee jeeeed zejejeejeed ZejdeDZe je jeeeeZe je jeee je j eddej!eej!ez ejeedZ"e je j e"ddej!ee"ej!e"z ejeZ#e#dde dz Z$e je j e$ddej!ee$ej!e$z eeZ%ee%&ee%'e%j(ee%)gdee%'gdejeedZejejeejeed ZeeZe je jeeejeedZ*e*eZ+e je+eej,ejeeejeddZ-e-eZ.e je.eededej!e+edej!edSdS)a from David Huard's scipy sandbox, also attached to a ticket and in the matplotlib-user mailinglist (links ???) Notes ===== out of bounds interpolation raises exception and would not be completely defined :: >>> scoreatpercentile(x, [0,25,50,100]) Traceback (most recent call last): ... raise ValueError("A value in x_new is below the interpolation " ValueError: A value in x_new is below the interpolation range. >>> percentileofscore(x, [-50, 50]) Traceback (most recent call last): ... raise ValueError("A value in x_new is below the interpolation " ValueError: A value in x_new is below the interpolation range. idea ==== histogram and empirical interpolated distribution ------------------------------------------------- dual constructor * empirical cdf : cdf on all observations through linear interpolation * binned cdf : based on histogram both should work essentially the same, although pdf of empirical has many spikes, fluctuates a lot - alternative: binning based on interpolated cdf : example in script * ppf: quantileatscore based on interpolated cdf * rvs : generic from ppf * stats, expectation ? how does integration wrt cdf work - theory? Problems * limits, lower and upper bound of support does not work or is undefined with empirical cdf and interpolation * extending bounds ? matlab has pareto tails for empirical distribution, breaks linearity empirical distribution with higher order interpolation ------------------------------------------------------ * should work easily enough with interpolating splines * not piecewise linear * can use pareto (or other) tails * ppf how do I get the inverse function of a higher order spline? Chuck: resample and fit spline to inverse function this will have an approximation error in the inverse function * -> does not work: higher order spline does not preserve monotonicity see mailing list for response to my question * pmf from derivative available in spline -> forget this and use kernel density estimator instead bootstrap/empirical distribution: --------------------------------- discrete distribution on real line given observations what's defined? * cdf : step function * pmf : points with equal weight 1/nobs * rvs : resampling * ppf : quantileatscore on sample? * moments : from data ? * expectation ? sum_{all observations x} [func(x) * pmf(x)] * similar for discrete distribution on real line * References : ? * what's the point? most of it is trivial, just for the record ? Created on Monday, May 03, 2010, 11:47:03 AM Author: josef-pktd, parts based on David Huard License: BSD Nctj|}t|}tjtj|tj|}||dz S)zReturn the score at the given percentile of the data. Example: >>> data = randn(100) >>> scoreatpercentile(data, 50) will return the median of sample `data`. Y@)nparray empiricalcdf interpolateinterp1dsort)data percentilepercdf interpolators f/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/statsmodels/sandbox/stats/stats_dhuard.pyscoreatpercentilerVsV (:  C t  C' bgdmmDDL <D ! !!ct|}tjtj|tj|}||dzS)aDReturn the percentile-position of score relative to data. score: Array of scores at which the percentile is computed. Return percentiles (0-100). Example r = randn(50) x = linspace(-2,2,100) percentileofscore(r,x) Raise an error if the score is outside the range of data. r)rrr rr )r scorerrs rpercentileofscorerdsH t  C' rws||DDL <  t ##rHazenctjtj|dz}t|}|}|dkr |dz |z }nc|dkr ||dzz }nT|dkr |dz |z }nE|dkr |dz |dzz }n3|d kr |dz |d zz }n!|d kr |d z |d zz }nt d|S)aReturn the empirical cdf. Methods available: Hazen: (i-0.5)/N Weibull: i/(N+1) Chegodayev: (i-.3)/(N+.4) Cunnane: (i-.4)/(N+.2) Gringorten: (i-.44)/(N+.12) California: (i-1)/N Where i goes from 1 to N. ?hazen?weibull california chegodayev333333?皙?cunnane皙? gringorten)\(?Q?[Unknown method. Choose among Weibull, Hazen,Chegodayev, Cunnane, Gringorten and California.)rargsortlenlower ValueError)r methodiNrs rrrvs 2:d##$$r)A D A \\^^F uai 9  2h <  tQh <  tadm 9  tadm <  uquoKLL L Jrc4eZdZdZdZd dZdZdZd d ZdS) HistDistzDistribution with piecewise linear cdf, pdf is step function can be created from empiricial distribution or from a histogram (not done yet) work in progress, not finished c(tj||_tj|j|jg|_tj|}|||_tj||_ | }tj ||_ tj|j|j |_tj|j |j|_dS)N)r atleast_1dr rminmaxbinlimitr& _datasortedrankingrr _empcdfsortedrr cdfintpppfintp)selfr sortindrs r__init__zHistDist.__init__sM$'' $)--//49==??!CDD *T""=z'** !!WS\\"+D,PQQ "+D,>@PQQ rNrc||j}|j}n)tjtj|dz}t |}|}|dkr |dz |z }nc|dkr ||dzz }nT|dkr |dz |z }nE|dkr |dz |d zz }n3|d kr |d z |d zz }n!|d kr |d z |dzz }nt d|S)aAReturn the empirical cdf. Methods available: Hazen: (i-0.5)/N Weibull: i/(N+1) Chegodayev: (i-.3)/(N+.4) Cunnane: (i-.4)/(N+.2) Gringorten: (i-.44)/(N+.12) California: (i-1)/N Where i goes from 1 to N. Nrrrrrrrrr r!r"r#r$r%)r r5rr&r'r(r))r9r r*r+r,rs rrzHistDist.empiricalcdfs <9D AA 2:d++,,r1A II W  S5!)CC y QrT(CC | # #R4(CC | # #R4!B$-CC y R4!B$-CC | # #S51S5/CCOPP P rc,||Sz& this is score in dh )r7)r9rs rcdf_empzHistDist.cdf_emps ||E"""rc,||Sr>)r8)r9quantiles rppf_empzHistDist.ppf_emps ||H%%%rFreedmancDt|j}|dkr7|d|dz }d|z|dzz}n(|dkr"dtj|jz|dzz}tj|j|z |_|jS)zFind the optimal number of bins and update the bin countaccordingly. Available methods : Freedman Scott rC??gUUUUUUտScottgQ @)r'r rBrstdptpr3nbin)r9r*nobsIQRwidths roptimize_binningzHistDist.optimize_binnings 49~~ :  ,,t$$t||D'9'99CsFD5M)EE W__26$),,,te}r_s)BBBb3??2..q1BBBr)rFrrE)ggпrrFrigQ?)rYsznegative densityz(np.diff(ppfs)).min()z(np.diff(cdf_ongrid)).min())r)/rSscipy.interpolaternumpyrrrrr.rPmatplotlib.pyplotpyplotpltrLrandomrandnxexamplesprintlinspacer1r2xsuppposplotInterpolatedUnivariateSpliner r[rpdfempfigure cdf_ongridstepdiffxsupp2xsoxshistdrOr?r3rBr8ppfsUnivariateSplineppfempppferTrrrsQQd(''''' " " "$$$$!!!!H````````H z###### D AsHH}} Q 3''((( 2&&''' AEEGGQUUWW--5)) ""1c**CE222 5K 4WRWQZZ UV@X@X[\ ] ] ]BBEBBBCC vSZZ   ###  ssGBGJ//>???QUUWWaeeggr22 WRWSS[[11'"'&//ABBBbgajj 47^ CRCR))'"'"++5666 HQKKE E% " "### E%-- ' '((( E%--))) * *+++ E%--333 4 4555 BK# . .E0 0GBGLLQROO