_Mhg XddlZddlZddlmZmZmZddlmZddl m Z m Z ddl Z ddl mZmZmZmZddlmZmZmZddlmZmZmZmZdd lmZdd lmZmZdd l m!Z!gd Z"d Z#dBdZ$dCdZ%dZ&dZ'dZ(dZ)e GddZ*edddddddddddd dZ+dZ,e Gdd Z-ed!d"dddddd#d$Z.e Gd%d&Z/d'Z0d(Z1d)dd*ddd+d,Z2e Gd-d.Z3d/Z4d0Z5d1Z6 dBd2Z7 dBd3Z8 dBd4Z9d5Z:edd6ddddddd7d8Z;e Gd9d:Ze Gd>d?e<Z?e Gd@dAe<Z@dS)DN) combinations permutationsproduct)Sequence) dataclassfield)check_random_state_rename_parameter rng_integers_transition_to_rng)array_namespaceis_numpyxp_moveaxis_to_end)ndtrndtricomb factorial)ConfidenceInterval)_broadcast_concatenate_broadcast_arrays)DegenerateDataWarning) bootstrapmonte_carlo_testpermutation_testcddfd }|S)zVectorize an n-sample statisticraxiscfd|D}tj|ddt|}tj|d}fd}tj|d|dS)Nc*g|]}|jSshape).0samplers W/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/scipy/stats/_resampling.py z9_vectorize_statistic..stat_nd..s 999&6<%999rc8tj|}|SN)npsplit)zdata split_indices statistics r&stat_1dz6_vectorize_statistic..stat_nd..stat_1d"s!8A}--D9d# #r(r!)r,cumsumrmoveaxisapply_along_axis)rr/lengthsr.r2r0r1s` @r&stat_ndz%_vectorize_statistic..stat_nds9999D999 '**3B3/ "4 . . K4 # # $ $ $ $ $ $"7Aq11"55r(r!)r1r7s` r&_vectorize_statisticr8s26666666 Nr(c#K|jd}|p|}td||D]}t|||z }tj||ft }tj|dd|||zfdtj|}tj|||f}|| ||dz f}|d|f}|VdS)z=Jackknife resample the sample. Only one-sample stats for now.r)rdtypeNFr.) r#rangeminr,onesbool fill_diagonalarange broadcast_toreshape) r%batchn batch_nominalk batch_actualji resampless r&_jackknife_resamplerL*s RAJQM 1a ' '  =!A#..  G\1%T 2 2 2 111a,../777 IaLL OA a0 1 1 aDLL,!, - -36N   r(c\|jd}t|d|||f}|d|f}|S)zBootstrap resample the sample.r)r.)r#r )r% n_resamplesrngrErJrKs r&_bootstrap_resamplerP>s; RA S!Qa 011AsAvI r(c|j|}||k|||k|zd|zz S)zVectorized, simplified `scipy.stats.percentileofscore`. Uses logic of the 'mean' value of percentileofscore's kind parameter. Unlike `stats.percentileofscore`, the percentile returned is a fraction in [0, 1]. r)r#sum)ascorerBs r&_percentile_of_scorerWIsJ  A YOOO & &!u*)9)9t)9)D)D DQ OOr(c|jdd}tj||}tj|tj}tj|D]n\}}tj|r5d}tjt|dtj ||<N||}tj ||||<o|dS)z9`np.percentile` with different percentile for each slice.Nr)r:zThe BCa confidence interval cannot be calculated. This problem is known to occur when the distribution is degenerate or the statistic is np.min. stacklevelr!) r#r,rB zeros_likefloat64 ndenumerateisnanwarningswarnrnan percentile) theta_hat_balphar# percentilesindicesalpha_imsg theta_hat_b_is r&_percentile_along_axisrkTs  crc "E OE5 ) )E-RZ888KN511 I I 8G   I=  M/44 C C C C#%6K '0M#%=#H#HK r?r(c ^tj||d|id}t||d}t|}g} t |D]l\} } d|D} g} t | |D]3}|| | <t | d}| ||ddi4| | md| D} d| D}d| D}d t|| |D}d t||D}d t||D}d t|zt|d zz }t|}| }||z}t||d||zz z z}||z}t||d||zz z z}|||fS)z(Bias-corrected and accelerated interval.r).Nr)rc8g|]}tj|dS))r, expand_dimsr$r%s r&r'z!_bca_interval..}s$AAA&2>&"--AAAr(c:g|]}tj|dS)r)r)r, concatenater$ theta_hat_is r&r'z!_bca_interval..s7555#N;R888555r(c(g|]}|jdSr)r"rss r&r'z!_bca_interval..s A A A[; R A A Ar(c<g|]}|ddS)r)T)rkeepdims)meanrss r&r'z!_bca_interval..s;888&#''R$'??888r(c,g|]\}}}|dz ||z zSrr!)r$ theta_hat_dotrtrEs r&r'z!_bca_interval..s< 8 8 8- {AU}{2 3 8 8 8r(cRg|]$\}}|dzd|dzz %S)rYr)rrSr$U_irEs r&r'z!_bca_interval..7 D D D63S!VLLbL ! !!Q$ & D D Dr(cRg|]$\}}|dzd|dzz %S)rRr)rr~rs r&r'z!_bca_interval..rr(gUUUUUU?g?r) r,asarrayrWr enumeraterLrappendziprSr)r/r1rrerdrD theta_hatrcz0_hat theta_hat_jirIr%samplesrtjackknife_sample broadcastedn_jtheta_hat_j_dotU_jinumsdensa_hatz_alphaz_1alphanum1alpha_1num2alpha_2s r& _bca_intervalrls>  99d66677 BI%k92FFFJ :  FLt__ ) ) 6 BADAAA 3FE B B A A )GAJ+G"===K   yy+?B?? @ @ @ @K((((55'3555L B AL A A AC88*6888O 8 8?L#66 8 8 8D E DSs^^ D D DD D DSs^^ D D DD #d))Oc$ii#. .EEllGxH G D6D!eDj.1122G H D6D!eDj.1122G GU ""r(c h|dvrtd|dtj|jv}|st |}t |} || krtdd} t |} n#t$rtdwxYw| dkrtdd }d |D}t}|D]M}t|j }| || t|Nt |d krtj|t d g}|D]M}|j | d krtdt#j|| d}||N|dvrtd|r^|dj d}|d dD]$}|j d|krd}t|%d||fd}t#j|g}t+|}|}hd}||vrtd|t |}||ks|dkrtd||}n*t |}||ks|dkrtdhd}| } | |vrtd|d}| t/| dst|d}| r | jjs|dkrt|t5| } ||||| ||||| | | f S)z5Input validation and standardization for `bootstrap`.FNT0`vectorized` must be `True`, `False`, or `None`.Nr`axis` must be an integer.rz%`data` must be a sequence of samples.z(`data` must contain at least one sample.aiIgnoring the dimension specified by `axis`, arrays in `data` do not have the same shape. Beginning in SciPy 1.16.0, `bootstrap` will explicitly broadcast elements of `data` to the same shape (ignoring `axis`) before performing the calculation. To avoid this warning in the meantime, ensure that all samples have the same shape (except potentially along `axis`).c6g|]}tj|Sr!)r, atleast_1drps r&r'z!_bootstrap_iv..s" 5 5 5fBM& ! ! 5 5 5r(rrYrZIeach sample in `data` must contain two or more observations along `axis`.r)>FTz#`paired` must be `True` or `False`.zIWhen `paired is True`, all samples must have the same length along `axis`c0fd|D}||d|iS)Nc$g|] }|df S).r!)r$r%rJs r&r'z4_bootstrap_iv..statistic..s!666vF36N666r(rr!)rJrr/unpaired_statistics` r&r1z _bootstrap_iv..statistics26666666D%%t7$77 7r(>lessgreater two-sidedz`alternative` must be one of z-`n_resamples` must be a non-negative integer.+`batch` must be a positive integer or None.>bcabasicrcz`method` must be in z?`bootstrap_result` must have attribute `bootstrap_distribution'bootstrap_distributionzXEither `bootstrap_result.bootstrap_distribution.size` or `n_resamples` must be positive.) ValueErrorinspect signature parametersr8intlen TypeErrorsetlistr#popaddtupler`ra FutureWarningr,r4rrAfloatlowerhasattrrsizer )r/r1 vectorizedpairedrconfidence_level alternativerNrDmethodbootstrap_resultrOaxis_int n_samplesmessagereduced_shapesr% reduced_shapedata_ivrEconfidence_level_float alternativesn_resamples_intbatch_ivmethodss r& _bootstrap_ivrs ,,,KLLLw0;;FF 4(33 4yyH x5666IBII BBB@AAABA~~CDDD,G 6 5 5 5 5DUUN11V\** $5//0000 >a g};;;;G < !Q & &:;; ;VXr22v ]"">??? ! AJ R abbk * *F|B1$$6 )))%!w9 8 8 8 89Q<<."#344##%%K333L,&&GGGHHH+&&Oo%%1)<)<HIII }u:: H  A JKK K,,,G \\^^F W999:::OG$,.FGG %!!!1G "  4 9"1$$!!! S ! !C Y FH "K( $c ++s )A99BcLeZdZUdZeed<ejed<eejzed<dS)BootstrapResultaResult object returned by `scipy.stats.bootstrap`. Attributes ---------- confidence_interval : ConfidenceInterval The bootstrap confidence interval as an instance of `collections.namedtuple` with attributes `low` and `high`. bootstrap_distribution : ndarray The bootstrap distribution, that is, the value of `statistic` for each resample. The last dimension corresponds with the resamples (e.g. ``res.bootstrap_distribution.shape[-1] == n_resamples``). standard_error : float or ndarray The bootstrap standard error, that is, the sample standard deviation of the bootstrap distribution. confidence_intervalrstandard_errorN) __name__ __module__ __qualname____doc__r__annotations__r,ndarrayrr!r(r&rrsK ,+++J&&&BJ&&&&&&r(r random_state'Fgffffff?rBCa) rNrDrrrrrrrrOc t|||||||||| | | } | \ }}}}}}}}}} } } | gn| jg} |p|pd}td||D]^}t|||z }g}|D])}t ||| }||*| ||ddi_t j| d} |dkrd|z d z nd|z }| d kr%t||d|| | dd }t}n |d|z f}d }|| |dd z}|| |dd z}| dkr||ddi}d |z|z d |z|z }}|dkr!t j |t j }n%|dkrt j |t j }tt||| t j| ddS)a9 Compute a two-sided bootstrap confidence interval of a statistic. When `method` is ``'percentile'`` and `alternative` is ``'two-sided'``, a bootstrap confidence interval is computed according to the following procedure. 1. Resample the data: for each sample in `data` and for each of `n_resamples`, take a random sample of the original sample (with replacement) of the same size as the original sample. 2. Compute the bootstrap distribution of the statistic: for each set of resamples, compute the test statistic. 3. Determine the confidence interval: find the interval of the bootstrap distribution that is - symmetric about the median and - contains `confidence_level` of the resampled statistic values. While the ``'percentile'`` method is the most intuitive, it is rarely used in practice. Two more common methods are available, ``'basic'`` ('reverse percentile') and ``'BCa'`` ('bias-corrected and accelerated'); they differ in how step 3 is performed. If the samples in `data` are taken at random from their respective distributions :math:`n` times, the confidence interval returned by `bootstrap` will contain the true value of the statistic for those distributions approximately `confidence_level`:math:`\, \times \, n` times. Parameters ---------- data : sequence of array-like Each element of `data` is a sample containing scalar observations from an underlying distribution. Elements of `data` must be broadcastable to the same shape (with the possible exception of the dimension specified by `axis`). .. versionchanged:: 1.14.0 `bootstrap` will now emit a ``FutureWarning`` if the shapes of the elements of `data` are not the same (with the exception of the dimension specified by `axis`). Beginning in SciPy 1.16.0, `bootstrap` will explicitly broadcast the elements to the same shape (except along `axis`) before performing the calculation. statistic : callable Statistic for which the confidence interval is to be calculated. `statistic` must be a callable that accepts ``len(data)`` samples as separate arguments and returns the resulting statistic. If `vectorized` is set ``True``, `statistic` must also accept a keyword argument `axis` and be vectorized to compute the statistic along the provided `axis`. n_resamples : int, default: ``9999`` The number of resamples performed to form the bootstrap distribution of the statistic. batch : int, optional The number of resamples to process in each vectorized call to `statistic`. Memory usage is O( `batch` * ``n`` ), where ``n`` is the sample size. Default is ``None``, in which case ``batch = n_resamples`` (or ``batch = max(n_resamples, n)`` for ``method='BCa'``). vectorized : bool, optional If `vectorized` is set ``False``, `statistic` will not be passed keyword argument `axis` and is expected to calculate the statistic only for 1D samples. If ``True``, `statistic` will be passed keyword argument `axis` and is expected to calculate the statistic along `axis` when passed an ND sample array. If ``None`` (default), `vectorized` will be set ``True`` if ``axis`` is a parameter of `statistic`. Use of a vectorized statistic typically reduces computation time. paired : bool, default: ``False`` Whether the statistic treats corresponding elements of the samples in `data` as paired. If True, `bootstrap` resamples an array of *indices* and uses the same indices for all arrays in `data`; otherwise, `bootstrap` independently resamples the elements of each array. axis : int, default: ``0`` The axis of the samples in `data` along which the `statistic` is calculated. confidence_level : float, default: ``0.95`` The confidence level of the confidence interval. alternative : {'two-sided', 'less', 'greater'}, default: ``'two-sided'`` Choose ``'two-sided'`` (default) for a two-sided confidence interval, ``'less'`` for a one-sided confidence interval with the lower bound at ``-np.inf``, and ``'greater'`` for a one-sided confidence interval with the upper bound at ``np.inf``. The other bound of the one-sided confidence intervals is the same as that of a two-sided confidence interval with `confidence_level` twice as far from 1.0; e.g. the upper bound of a 95% ``'less'`` confidence interval is the same as the upper bound of a 90% ``'two-sided'`` confidence interval. method : {'percentile', 'basic', 'bca'}, default: ``'BCa'`` Whether to return the 'percentile' bootstrap confidence interval (``'percentile'``), the 'basic' (AKA 'reverse') bootstrap confidence interval (``'basic'``), or the bias-corrected and accelerated bootstrap confidence interval (``'BCa'``). bootstrap_result : BootstrapResult, optional Provide the result object returned by a previous call to `bootstrap` to include the previous bootstrap distribution in the new bootstrap distribution. This can be used, for example, to change `confidence_level`, change `method`, or see the effect of performing additional resampling without repeating computations. rng : `numpy.random.Generator`, optional Pseudorandom number generator state. When `rng` is None, a new `numpy.random.Generator` is created using entropy from the operating system. Types other than `numpy.random.Generator` are passed to `numpy.random.default_rng` to instantiate a ``Generator``. Returns ------- res : BootstrapResult An object with attributes: confidence_interval : ConfidenceInterval The bootstrap confidence interval as an instance of `collections.namedtuple` with attributes `low` and `high`. bootstrap_distribution : ndarray The bootstrap distribution, that is, the value of `statistic` for each resample. The last dimension corresponds with the resamples (e.g. ``res.bootstrap_distribution.shape[-1] == n_resamples``). standard_error : float or ndarray The bootstrap standard error, that is, the sample standard deviation of the bootstrap distribution. Warns ----- `~scipy.stats.DegenerateDataWarning` Generated when ``method='BCa'`` and the bootstrap distribution is degenerate (e.g. all elements are identical). Notes ----- Elements of the confidence interval may be NaN for ``method='BCa'`` if the bootstrap distribution is degenerate (e.g. all elements are identical). In this case, consider using another `method` or inspecting `data` for indications that other analysis may be more appropriate (e.g. all observations are identical). References ---------- .. [1] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap, Chapman & Hall/CRC, Boca Raton, FL, USA (1993) .. [2] Nathaniel E. Helwig, "Bootstrap Confidence Intervals", http://users.stat.umn.edu/~helwig/notes/bootci-Notes.pdf .. [3] Bootstrapping (statistics), Wikipedia, https://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29 Examples -------- Suppose we have sampled data from an unknown distribution. >>> import numpy as np >>> rng = np.random.default_rng() >>> from scipy.stats import norm >>> dist = norm(loc=2, scale=4) # our "unknown" distribution >>> data = dist.rvs(size=100, random_state=rng) We are interested in the standard deviation of the distribution. >>> std_true = dist.std() # the true value of the statistic >>> print(std_true) 4.0 >>> std_sample = np.std(data) # the sample statistic >>> print(std_sample) 3.9460644295563863 The bootstrap is used to approximate the variability we would expect if we were to repeatedly sample from the unknown distribution and calculate the statistic of the sample each time. It does this by repeatedly resampling values *from the original sample* with replacement and calculating the statistic of each resample. This results in a "bootstrap distribution" of the statistic. >>> import matplotlib.pyplot as plt >>> from scipy.stats import bootstrap >>> data = (data,) # samples must be in a sequence >>> res = bootstrap(data, np.std, confidence_level=0.9, rng=rng) >>> fig, ax = plt.subplots() >>> ax.hist(res.bootstrap_distribution, bins=25) >>> ax.set_title('Bootstrap Distribution') >>> ax.set_xlabel('statistic value') >>> ax.set_ylabel('frequency') >>> plt.show() The standard error quantifies this variability. It is calculated as the standard deviation of the bootstrap distribution. >>> res.standard_error 0.24427002125829136 >>> res.standard_error == np.std(res.bootstrap_distribution, ddof=1) True The bootstrap distribution of the statistic is often approximately normal with scale equal to the standard error. >>> x = np.linspace(3, 5) >>> pdf = norm.pdf(x, loc=std_sample, scale=res.standard_error) >>> fig, ax = plt.subplots() >>> ax.hist(res.bootstrap_distribution, bins=25, density=True) >>> ax.plot(x, pdf) >>> ax.set_title('Normal Approximation of the Bootstrap Distribution') >>> ax.set_xlabel('statistic value') >>> ax.set_ylabel('pdf') >>> plt.show() This suggests that we could construct a 90% confidence interval on the statistic based on quantiles of this normal distribution. >>> norm.interval(0.9, loc=std_sample, scale=res.standard_error) (3.5442759991341726, 4.3478528599786) Due to central limit theorem, this normal approximation is accurate for a variety of statistics and distributions underlying the samples; however, the approximation is not reliable in all cases. Because `bootstrap` is designed to work with arbitrary underlying distributions and statistics, it uses more advanced techniques to generate an accurate confidence interval. >>> print(res.confidence_interval) ConfidenceInterval(low=3.57655333533867, high=4.382043696342881) If we sample from the original distribution 100 times and form a bootstrap confidence interval for each sample, the confidence interval contains the true value of the statistic approximately 90% of the time. >>> n_trials = 100 >>> ci_contains_true_std = 0 >>> for i in range(n_trials): ... data = (dist.rvs(size=100, random_state=rng),) ... res = bootstrap(data, np.std, confidence_level=0.9, ... n_resamples=999, rng=rng) ... ci = res.confidence_interval ... if ci[0] < std_true < ci[1]: ... ci_contains_true_std += 1 >>> print(ci_contains_true_std) 88 Rather than writing a loop, we can also determine the confidence intervals for all 100 samples at once. >>> data = (dist.rvs(size=(n_trials, 100), random_state=rng),) >>> res = bootstrap(data, np.std, axis=-1, confidence_level=0.9, ... n_resamples=999, rng=rng) >>> ci_l, ci_u = res.confidence_interval Here, `ci_l` and `ci_u` contain the confidence interval for each of the ``n_trials = 100`` samples. >>> print(ci_l[:5]) [3.86401283 3.33304394 3.52474647 3.54160981 3.80569252] >>> print(ci_u[:5]) [4.80217409 4.18143252 4.39734707 4.37549713 4.72843584] And again, approximately 90% contain the true value, ``std_true = 4``. >>> print(np.sum((ci_l < std_true) & (std_true < ci_u))) 93 `bootstrap` can also be used to estimate confidence intervals of multi-sample statistics. For example, to get a confidence interval for the difference between means, we write a function that accepts two sample arguments and returns only the statistic. The use of the ``axis`` argument ensures that all mean calculations are perform in a single vectorized call, which is faster than looping over pairs of resamples in Python. >>> def my_statistic(sample1, sample2, axis=-1): ... mean1 = np.mean(sample1, axis=axis) ... mean2 = np.mean(sample2, axis=axis) ... return mean1 - mean2 Here, we use the 'percentile' method with the default 95% confidence level. >>> sample1 = norm.rvs(scale=1, size=100, random_state=rng) >>> sample2 = norm.rvs(scale=2, size=100, random_state=rng) >>> data = (sample1, sample2) >>> res = bootstrap(data, my_statistic, method='basic', rng=rng) >>> print(my_statistic(sample1, sample2)) 0.16661030792089523 >>> print(res.confidence_interval) ConfidenceInterval(low=-0.29087973240818693, high=0.6371338699912273) The bootstrap estimate of the standard error is also available. >>> print(res.standard_error) 0.238323948262459 Paired-sample statistics work, too. For example, consider the Pearson correlation coefficient. >>> from scipy.stats import pearsonr >>> n = 100 >>> x = np.linspace(0, 10, n) >>> y = x + rng.uniform(size=n) >>> print(pearsonr(x, y)[0]) # element 0 is the statistic 0.9954306665125647 We wrap `pearsonr` so that it returns only the statistic, ensuring that we use the `axis` argument because it is available. >>> def my_statistic(x, y, axis=-1): ... return pearsonr(x, y, axis=axis)[0] We call `bootstrap` using ``paired=True``. >>> res = bootstrap((x, y), my_statistic, paired=True, rng=rng) >>> print(res.confidence_interval) ConfidenceInterval(low=0.9941504301315878, high=0.996377412215445) The result object can be passed back into `bootstrap` to perform additional resampling: >>> len(res.bootstrap_distribution) 9999 >>> res = bootstrap((x, y), my_statistic, paired=True, ... n_resamples=1000, rng=rng, ... bootstrap_result=res) >>> len(res.bootstrap_distribution) 10999 or to change the confidence interval options: >>> res2 = bootstrap((x, y), my_statistic, paired=True, ... n_resamples=0, rng=rng, bootstrap_result=res, ... method='percentile', confidence_level=0.9) >>> np.testing.assert_equal(res2.bootstrap_distribution, ... res.bootstrap_distribution) >>> res.confidence_interval ConfidenceInterval(low=0.9941574828235082, high=0.9963781698210212) without repeating computation of the original bootstrap distribution. Nrr)rNrOrr)rrrRr)rrerdrDc0tj||dS)Nr))rTqr)r,rc)rTrs r&percentile_funz!bootstrap..percentile_funs=1333 3r(drrr)ddofr)rrr)rrr<r=rPrr,rrrrk full_likeinfrrstd)r/r1rNrDrrrrrrrrOargsrdrFrGrHresampled_datar%resamplereintervalrci_lci_urs r&rr%s^ y*fd); U!13 8 8D  T9j&$0@+uf.>)122)@A-[-AM 1k= 1 1 @ @=+a-88  , ,F*6|/2444H  ! !( + + + + 99n>2>>????.2666K*5 )C)Ca""A % %''   yr-8GGGGIrK/!E'> 4 4 4 >+x{3 7 7D >+x{3 7 7D It-"-- y[4'9t);df|D26'**  ! !|D"&)) /A$/M/M2=*,&12*N*N*N P P PPr(c Dt|}||krtd|dvrtdt|ts|f}|f}|D] } t | st d!d} t |n"#t $r} t| | d} ~ wwxYwt |t |kst| t |st d|G tj|j } n(#t$r} d|d } t| | d} ~ wwxYwd | v}t|} |s9t| rt|}nd | j d } t| |}t||| }g}|D]L}|jdkr| |dn|}t#||| }||Mt|}||ks|dkrtd||}n*t|}||ks|dkrtdhd}|}||vrtd|t)| d| j}| jg||R}|||||||||| f S)(Input validation for `monte_carlo_test`.rrr0`rvs` must be callable or sequence of callables.z:If `rvs` is a sequence, `len(rvs)` must equal `len(data)`.Nz`statistic` must be callable.z"Signature inspection of statistic=z% failed; pass `vectorize` explicitly.rzV`statistic` must be vectorized (i.e. support an `axis` argument) when `data` contains z arrays.)xprr{)`n_resamples` must be a positive integer.r>rrr`alternative` must be in float16)rr isinstancercallablerrrrrr rr8rrndimrBrrrgetattrfloat32 result_type)r/rvsr1rrNrDrrrrvs_irerrstatistic_vectorizedrr%rrr min_floatr;s r&_monte_carlo_test_ivrsl4yyH x5666,,,KLLL c8 $ $fwPP PNOO O P KG) D )))!!q() s88s4yy !!! I  97888 -))44?II - - -6)666GW%%1 , -y( $ B ) B<<  #B#B O9;OOOGW%% %( T4B / / /DG28+2B2B...#FH<<<v+&&Oo%%A)=)=DEEE }u:: H  A JKK K333L##%%K,&&C\CCDDDIrz22I BN /G /Y / / /E S. O k8UB 88s09B B(B##B(;D D:D55D:c\eZdZUdZeejzed<eejzed<ejed<dS)MonteCarloTestResultatResult object returned by `scipy.stats.monte_carlo_test`. Attributes ---------- statistic : float or ndarray The observed test statistic of the sample. pvalue : float or ndarray The p-value for the given alternative. null_distribution : ndarray The values of the test statistic generated under the null hypothesis. r1pvaluenull_distributionNrrrrrr,rrr!r(r&rrR  rz!!!! BJ z!!!!!r(rr%r/)rrNrDrrc bt|||||||}|\ }}}}}}}||ddi} | jdkr| dn| } d|D} |p} g} td| D]L} t | | z fdt || D}| ||ddiM| } | dgdg| jzz}  | j dsdn! | j j d z} || zfd fd fd }|d }||| | }|dd}t| || S)aPerform a Monte Carlo hypothesis test. `data` contains a sample or a sequence of one or more samples. `rvs` specifies the distribution(s) of the sample(s) in `data` under the null hypothesis. The value of `statistic` for the given `data` is compared against a Monte Carlo null distribution: the value of the statistic for each of `n_resamples` sets of samples generated using `rvs`. This gives the p-value, the probability of observing such an extreme value of the test statistic under the null hypothesis. Parameters ---------- data : array-like or sequence of array-like An array or sequence of arrays of observations. rvs : callable or tuple of callables A callable or sequence of callables that generates random variates under the null hypothesis. Each element of `rvs` must be a callable that accepts keyword argument ``size`` (e.g. ``rvs(size=(m, n))``) and returns an N-d array sample of that shape. If `rvs` is a sequence, the number of callables in `rvs` must match the number of samples in `data`, i.e. ``len(rvs) == len(data)``. If `rvs` is a single callable, `data` is treated as a single sample. statistic : callable Statistic for which the p-value of the hypothesis test is to be calculated. `statistic` must be a callable that accepts a sample (e.g. ``statistic(sample)``) or ``len(rvs)`` separate samples (e.g. ``statistic(samples1, sample2)`` if `rvs` contains two callables and `data` contains two samples) and returns the resulting statistic. If `vectorized` is set ``True``, `statistic` must also accept a keyword argument `axis` and be vectorized to compute the statistic along the provided `axis` of the samples in `data`. vectorized : bool, optional If `vectorized` is set ``False``, `statistic` will not be passed keyword argument `axis` and is expected to calculate the statistic only for 1D samples. If ``True``, `statistic` will be passed keyword argument `axis` and is expected to calculate the statistic along `axis` when passed ND sample arrays. If ``None`` (default), `vectorized` will be set ``True`` if ``axis`` is a parameter of `statistic`. Use of a vectorized statistic typically reduces computation time. n_resamples : int, default: 9999 Number of samples drawn from each of the callables of `rvs`. Equivalently, the number statistic values under the null hypothesis used as the Monte Carlo null distribution. batch : int, optional The number of Monte Carlo samples to process in each call to `statistic`. Memory usage is O( `batch` * ``sample.size[axis]`` ). Default is ``None``, in which case `batch` equals `n_resamples`. alternative : {'two-sided', 'less', 'greater'} The alternative hypothesis for which the p-value is calculated. For each alternative, the p-value is defined as follows. - ``'greater'`` : the percentage of the null distribution that is greater than or equal to the observed value of the test statistic. - ``'less'`` : the percentage of the null distribution that is less than or equal to the observed value of the test statistic. - ``'two-sided'`` : twice the smaller of the p-values above. axis : int, default: 0 The axis of `data` (or each sample within `data`) over which to calculate the statistic. Returns ------- res : MonteCarloTestResult An object with attributes: statistic : float or ndarray The test statistic of the observed `data`. pvalue : float or ndarray The p-value for the given alternative. null_distribution : ndarray The values of the test statistic generated under the null hypothesis. .. warning:: The p-value is calculated by counting the elements of the null distribution that are as extreme or more extreme than the observed value of the statistic. Due to the use of finite precision arithmetic, some statistic functions return numerically distinct values when the theoretical values would be exactly equal. In some cases, this could lead to a large error in the calculated p-value. `monte_carlo_test` guards against this by considering elements in the null distribution that are "close" (within a relative tolerance of 100 times the floating point epsilon of inexact dtypes) to the observed value of the test statistic as equal to the observed value of the test statistic. However, the user is advised to inspect the null distribution to assess whether this method of comparison is appropriate, and if not, calculate the p-value manually. References ---------- .. [1] B. Phipson and G. K. Smyth. "Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn." Statistical Applications in Genetics and Molecular Biology 9.1 (2010). Examples -------- Suppose we wish to test whether a small sample has been drawn from a normal distribution. We decide that we will use the skew of the sample as a test statistic, and we will consider a p-value of 0.05 to be statistically significant. >>> import numpy as np >>> from scipy import stats >>> def statistic(x, axis): ... return stats.skew(x, axis) After collecting our data, we calculate the observed value of the test statistic. >>> rng = np.random.default_rng() >>> x = stats.skewnorm.rvs(a=1, size=50, random_state=rng) >>> statistic(x, axis=0) 0.12457412450240658 To determine the probability of observing such an extreme value of the skewness by chance if the sample were drawn from the normal distribution, we can perform a Monte Carlo hypothesis test. The test will draw many samples at random from their normal distribution, calculate the skewness of each sample, and compare our original skewness against this distribution to determine an approximate p-value. >>> from scipy.stats import monte_carlo_test >>> # because our statistic is vectorized, we pass `vectorized=True` >>> rvs = lambda size: stats.norm.rvs(size=size, random_state=rng) >>> res = monte_carlo_test(x, rvs, statistic, vectorized=True) >>> print(res.statistic) 0.12457412450240658 >>> print(res.pvalue) 0.7012 The probability of obtaining a test statistic less than or equal to the observed value under the null hypothesis is ~70%. This is greater than our chosen threshold of 5%, so we cannot consider this to be significant evidence against the null hypothesis. Note that this p-value essentially matches that of `scipy.stats.skewtest`, which relies on an asymptotic distribution of a test statistic based on the sample skewness. >>> stats.skewtest(x).pvalue 0.6892046027110614 This asymptotic approximation is not valid for small sample sizes, but `monte_carlo_test` can be used with samples of any size. >>> x = stats.skewnorm.rvs(a=1, size=7, random_state=rng) >>> # stats.skewtest(x) would produce an error due to small sample >>> res = monte_carlo_test(x, rvs, statistic, vectorized=True) The Monte Carlo distribution of the test statistic is provided for further investigation. >>> import matplotlib.pyplot as plt >>> fig, ax = plt.subplots() >>> ax.hist(res.null_distribution, bins=50) >>> ax.set_title("Monte Carlo distribution of test statistic") >>> ax.set_xlabel("Value of Statistic") >>> ax.set_ylabel("Frequency") >>> plt.show() rr)rr!c(g|]}|jdSrvr"rps r&r'z$monte_carlo_test..s:::6fl2&:::r(c2g|]\}}||fS)rr!)r$rn_observations_irHs r&r'z$monte_carlo_test..sGNNN0 0U/? @AAANNNr(rz real floatingrc||zk}|}|ddzdzz }|SNr:r)rr;?rrSrobservedcmpspvaluesr;gammarNrs r&rzmonte_carlo_test..lessR Hu$44zz$ez,,66$Qe644r9kB>NOr(c||z k}|}|ddzdzz }|Sr r r s r&rz!monte_carlo_test..greaterrr(cj||}||}||dz}|SNrR)minimum)rr  pvalues_lesspvalues_greaterrrrrs r& two_sidedz#monte_carlo_test..two_sidedsDt-x88 !'"3X>>**\?;;a?r(rrrgr )rrrr<r=rrconcatrCisdtyper;finfoepsabsclipr)r/rr1rrNrDrrrr n_observationsrFrrGrKrrcomparerrHr;rrrrs ` @@@@@@r&rrsP c9j +UK G GD-1*T3 :{ Kubzz))T33344H'}11x||xH::T:::N([M 1k= 1 1AA=+/:: NNNN47^4L4LNNN   I!?B!?!?@@@@ "344 #4rdaS=N6NOOZZBB 2AA(.))-c1 FF3> " "E   !%''G#gk"#4h??Ggggr2&&G '3D E EEr(cHeZdZUdZeejzed<eejzed<dS) PowerResultzResult object returned by `scipy.stats.power`. Attributes ---------- power : float or ndarray The estimated power. pvalues : float or ndarray The simulated p-values. powerrNrr!r(r&r#r#sD 2:  RZ r(r#c ttj|j}n#t $rdh}YnwxYw||dd}|S)z?Wrap callable to accept arbitrary kwargs and ignore unused onesr)keysfuncTfd|D}||i|S)Nc$i|] \}}|v || Sr!r!)r$keyvalr&s r& z7_wrap_kwargs..wrapped_rvs_i..s/"""xsCD[[s [[r()items)r&r'r all_kwargskwargss` r& wrapped_rvs_iz#_wrap_kwargs..wrapped_rvs_isK"""":+;+;+=+="""sD#F###r()rrrrr&r)r'r&r0s r& _wrap_kwargsr1s~7$S))499;;<< x #'C$$$$$ s8; A  A c |dvrtdt|ts|f}|f}|D] }t|st d!t |t |ksd} t| t j|d}t j|j t j r0t j |dkst j |dkrtd|tn|}t|tst d |} |} d |D} t jt jg|| R} | j}| jdkr| t jd d f} n"| |dd fj} | d d d t |f| d d t |d f} }|t.}t|st d |dt1j|jv}|st7|}n|}t9|}t/|}||ks|dkrtd||}n*t/|}||ks|dkrtd| ||||||| | |dd f S)rrrrzDIf `rvs` is a sequence, `len(rvs)` must equal `len(n_observations)`.r!rrz3`significance` must contain floats between 0 and 1.Nz;`kwargs` must be a dictionary that maps keywords to arrays.c,g|]}t|Sr!)r1)r$rs r&r'z_power_iv...s 8885<&&888r(r)z`test` must be callable.rrr)rrrrrrr,r issubdtyper;floatingr=maxdictvaluesr&broadcast_arraysr#rnewaxisrCTastyperrrrr8r1)rtestr  significancerrNrDr/rrvalsr& wrapped_rvstmpr#nobstest_vectorizedrrs r& _power_ivrDs,,,KLLL c8 $ $+f(*PP PNOO O P s88s>** * *7!!!:l++B/L M,,bk : :Pvl##a''26,+?+?!+C+CNOOO~TVVV6F fd # #WUVVV ==??D ;;==D98C888K *R(@.@4@@@ A AC IE x1}}"*aaa- kk58R.))+QQQ S \"C3s8899 $5$D ;;s  D D>>42333w066AA .t44"?33O+&&Oo%%A)=)=DEEE }u:: H  A JKK K $ j XtT59 >>r(g{Gz?i')r>rrNrDr/c xt||||||||}|\ }}} }}}}} } } |p|} g}t| | D]\}}tt| |g}td|| D]d}t | ||z fdt||D}||iddi}t |d|}||etj|d}||| |j ddz } tj|d}| | dz}|j dkrDtt|j |j |j z}tj ||}tj||kd}t|| S) a#Simulate the power of a hypothesis test under an alternative hypothesis. Parameters ---------- test : callable Hypothesis test for which the power is to be simulated. `test` must be a callable that accepts a sample (e.g. ``test(sample)``) or ``len(rvs)`` separate samples (e.g. ``test(samples1, sample2)`` if `rvs` contains two callables and `n_observations` contains two values) and returns the p-value of the test. If `vectorized` is set to ``True``, `test` must also accept a keyword argument `axis` and be vectorized to perform the test along the provided `axis` of the samples. Any callable from `scipy.stats` with an `axis` argument that returns an object with a `pvalue` attribute is also acceptable. rvs : callable or tuple of callables A callable or sequence of callables that generate(s) random variates under the alternative hypothesis. Each element of `rvs` must accept keyword argument ``size`` (e.g. ``rvs(size=(m, n))``) and return an N-d array of that shape. If `rvs` is a sequence, the number of callables in `rvs` must match the number of elements of `n_observations`, i.e. ``len(rvs) == len(n_observations)``. If `rvs` is a single callable, `n_observations` is treated as a single element. n_observations : tuple of ints or tuple of integer arrays If a sequence of ints, each is the sizes of a sample to be passed to `test`. If a sequence of integer arrays, the power is simulated for each set of corresponding sample sizes. See Examples. significance : float or array_like of floats, default: 0.01 The threshold for significance; i.e., the p-value below which the hypothesis test results will be considered as evidence against the null hypothesis. Equivalently, the acceptable rate of Type I error under the null hypothesis. If an array, the power is simulated for each significance threshold. kwargs : dict, optional Keyword arguments to be passed to `rvs` and/or `test` callables. Introspection is used to determine which keyword arguments may be passed to each callable. The value corresponding with each keyword must be an array. Arrays must be broadcastable with one another and with each array in `n_observations`. The power is simulated for each set of corresponding sample sizes and arguments. See Examples. vectorized : bool, optional If `vectorized` is set to ``False``, `test` will not be passed keyword argument `axis` and is expected to perform the test only for 1D samples. If ``True``, `test` will be passed keyword argument `axis` and is expected to perform the test along `axis` when passed N-D sample arrays. If ``None`` (default), `vectorized` will be set ``True`` if ``axis`` is a parameter of `test`. Use of a vectorized test typically reduces computation time. n_resamples : int, default: 10000 Number of samples drawn from each of the callables of `rvs`. Equivalently, the number tests performed under the alternative hypothesis to approximate the power. batch : int, optional The number of samples to process in each call to `test`. Memory usage is proportional to the product of `batch` and the largest sample size. Default is ``None``, in which case `batch` equals `n_resamples`. Returns ------- res : PowerResult An object with attributes: power : float or ndarray The estimated power against the alternative. pvalues : ndarray The p-values observed under the alternative hypothesis. Notes ----- The power is simulated as follows: - Draw many random samples (or sets of samples), each of the size(s) specified by `n_observations`, under the alternative specified by `rvs`. - For each sample (or set of samples), compute the p-value according to `test`. These p-values are recorded in the ``pvalues`` attribute of the result object. - Compute the proportion of p-values that are less than the `significance` level. This is the power recorded in the ``power`` attribute of the result object. Suppose that `significance` is an array with shape ``shape1``, the elements of `kwargs` and `n_observations` are mutually broadcastable to shape ``shape2``, and `test` returns an array of p-values of shape ``shape3``. Then the result object ``power`` attribute will be of shape ``shape1 + shape2 + shape3``, and the ``pvalues`` attribute will be of shape ``shape2 + shape3 + (n_resamples,)``. Examples -------- Suppose we wish to simulate the power of the independent sample t-test under the following conditions: - The first sample has 10 observations drawn from a normal distribution with mean 0. - The second sample has 12 observations drawn from a normal distribution with mean 1.0. - The threshold on p-values for significance is 0.05. >>> import numpy as np >>> from scipy import stats >>> rng = np.random.default_rng(2549598345528) >>> >>> test = stats.ttest_ind >>> n_observations = (10, 12) >>> rvs1 = rng.normal >>> rvs2 = lambda size: rng.normal(loc=1, size=size) >>> rvs = (rvs1, rvs2) >>> res = stats.power(test, rvs, n_observations, significance=0.05) >>> res.power 0.6116 With samples of size 10 and 12, respectively, the power of the t-test with a significance threshold of 0.05 is approximately 60% under the chosen alternative. We can investigate the effect of sample size on the power by passing sample size arrays. >>> import matplotlib.pyplot as plt >>> nobs_x = np.arange(5, 21) >>> nobs_y = nobs_x >>> n_observations = (nobs_x, nobs_y) >>> res = stats.power(test, rvs, n_observations, significance=0.05) >>> ax = plt.subplot() >>> ax.plot(nobs_x, res.power) >>> ax.set_xlabel('Sample Size') >>> ax.set_ylabel('Simulated Power') >>> ax.set_title('Simulated Power of `ttest_ind` with Equal Sample Sizes') >>> plt.show() Alternatively, we can investigate the impact that effect size has on the power. In this case, the effect size is the location of the distribution underlying the second sample. >>> n_observations = (10, 12) >>> loc = np.linspace(0, 1, 20) >>> rvs2 = lambda size, loc: rng.normal(loc=loc, size=size) >>> rvs = (rvs1, rvs2) >>> res = stats.power(test, rvs, n_observations, significance=0.05, ... kwargs={'loc': loc}) >>> ax = plt.subplot() >>> ax.plot(loc, res.power) >>> ax.set_xlabel('Effect Size') >>> ax.set_ylabel('Simulated Power') >>> ax.set_title('Simulated Power of `ttest_ind`, Varying Effect Size') >>> plt.show() We can also use `power` to estimate the Type I error rate (also referred to by the ambiguous term "size") of a test and assess whether it matches the nominal level. For example, the null hypothesis of `jarque_bera` is that the sample was drawn from a distribution with the same skewness and kurtosis as the normal distribution. To estimate the Type I error rate, we can consider the null hypothesis to be a true *alternative* hypothesis and calculate the power. >>> test = stats.jarque_bera >>> n_observations = 10 >>> rvs = rng.normal >>> significance = np.linspace(0.0001, 0.1, 1000) >>> res = stats.power(test, rvs, n_observations, significance=significance) >>> size = res.power As shown below, the Type I error rate of the test is far below the nominal level for such a small sample, as mentioned in its documentation. >>> ax = plt.subplot() >>> ax.plot(significance, size) >>> ax.plot([0, 0.1], [0, 0.1], '--') >>> ax.set_xlabel('nominal significance level') >>> ax.set_ylabel('estimated test size (Type I error rate)') >>> ax.set_title('Estimated test size vs nominal significance level') >>> ax.set_aspect('equal', 'box') >>> ax.legend(('`ttest_1samp`', 'ideal test')) >>> plt.show() As one might expect from such a conservative test, the power is quite low with respect to some alternatives. For example, the power of the test under the alternative that the sample was drawn from the Laplace distribution may not be much greater than the Type I error rate. >>> rvs = rng.laplace >>> significance = np.linspace(0.0001, 0.1, 1000) >>> res = stats.power(test, rvs, n_observations, significance=0.05) >>> print(res.power) 0.0587 This is not a mistake in SciPy's implementation; it is simply due to the fact that the null distribution of the test statistic is derived under the assumption that the sample size is large (i.e. approaches infinity), and this asymptotic approximation is not accurate for small samples. In such cases, resampling and Monte Carlo methods (e.g. `permutation_test`, `goodness_of_fit`, `monte_carlo_test`) may be more appropriate. rc.g|]\}}|dd|fiS)rr!r!)r$rvs_jnobs_ijrHkwargs_is r&r'zpower..%sMAAA+UGHH\7$;HxHHAAAr(rr)rrNrv)r$r)rDrr7r<r=rrr,rrr#rCrrroryr#)r=rr r>rrNrDr/rArBrkwdsr#rFrnobs_iargs_i pvalues_irGrKrespnewdimspowersrHrIs @@r&r$r$WsD C~| UF < >LAAAAA/23/?/?AAAI$ 7X77B777CXs++A   Q    N92666 y!!!! Y_SbS !!EnW1---Gooeem,,G1l/ @Q1QRRSS~lG<< WW|+" 5 5 5F VW 5 5 55r(c\eZdZUdZeejzed<eejzed<ejed<dS)PermutationTestResultarResult object returned by `scipy.stats.permutation_test`. Attributes ---------- statistic : float or ndarray The observed test statistic of the data. pvalue : float or ndarray The p-value for the given alternative. null_distribution : ndarray The values of the test statistic generated under the null hypothesis. r1rrNrr!r(r&rSrS;rr(rSc#Kdfdtttj|}||ddD]<}tjd|Dt }|V=dS)zu Generate all partitions of indices of groups of given sizes, concatenated `ns` is an iterable of ints. c3dKt||D]}t|}||z }||gVdSr+)rr)r.rEcx0x1s r&all_partitionsz4_all_partitions_concatenated..all_partitionsTsLa##  AQBRBr(NNNN  r(c3Kt|dkr|gVdS||dD].}|d|ddD]}|dd|zV/dS)Nrr)r)r.nsrVdrYall_partitions_ns r&r]z6_all_partitions_concatenated..all_partitions_nZs r77a<<#III F2a5)) ! !A%%adBqrrF33 ! !!fqj     ! ! !r(Nc,g|]}t|Sr!)r)r$ partitions r&r'z0_all_partitions_concatenated..ds.;;; )!OO;;;r()rr<r,rSrrr<r)r[r. partitioningxrYr]s @@r&_all_partitions_concatenatedrbNs  !!!!!! E"&**  A((BqqqE22 N;;-9;;; < <.ns999'!T999r(cg|]\}}|Sr!r!res r&r'z$_batch_generator..qs = = =gaT = = =r(N)iterrrr<)iterablerDiteratorr.s r&_batch_generatorrkisH~~H zz455599Sux88999A > = =U5\\8!>>>>r(cttdr fd}n fd}|S)Npermutedc3Ktj}tj|df}tdD]8}t |z }|d}|d|V9dS)Nrrr)r)r,rAtiler<r=rm) rgrGrHpermuted_indicesrD n_obs_samplen_permutationsrrOs r&batched_perm_generatorz:_pairings_permutations_gen..batched_perm_generator|si --Gggy!'<==G1ne44 6 6"5.*:;; #&<<b<#A#A &} }55555  6 6r(c3KtdD]P}t|z }|f}|}tj|dd|VQdS)Nrrr)r)r<r=randomr,argsort) rGrHrrarDrqrrrrOs r&rsz:_pairings_permutations_gen..batched_perm_generators1ne44 < <"5.*:;; $i>JJDJ))j,,,]l];;;;;  < .s333Fv|B333r(r)cNg|]!}t||dz "Sr{)r)r$rJn_obs_ics r&r'z(_calculate_null_both..s?999(1+x!}55999r(rrTFc3BK|]}VdSr+) permutation)r$rJn_obsrOs r& z'_calculate_null_both..sC:://%00::::::r(rrD.rnNr) rr,r3prodr<rbrrrrkarrayr4r-r)r/r1rrrDrOrn_obs_in_max exact_testperm_generatorrrg data_batchr~r{s ` @@r&_calculate_null_bothrs D I43d333Gy!!H RLE G9999#IaKB77999 : :E  5g>> :::::#(#8#8:::  (S((E >$R ( ( (D#N%@@@BB(7###w,' [R33 Xj(3B3-bAAA   J!@R!@!@AAAA'8qAAA nj 88r(ct|}|djdt|z}||krId}|}|pt|}t fdt |D}t ||} n#d}|pt|}||||f} t| } g} | D]} tj | } tj | dd} dg|z} t |D]:}||d | |f| |<tj | |d d| |<;| || d ditj | d } | ||fS) z< Calculate null distribution for association tests. rr)Tc3PK|] }ttV!dSr+)rr<)r$rJrqs r&rz+_calculate_null_pairings..sG#>#>'($0l0C0C#D#D#>#>#>#>#>#>r(rFrN.rnrr)rr#rrrr<rkrwr,rswapaxesr4rrr)r/r1rrrDrOrrrrrsrrrgrrJrqs @r&_calculate_null_pairingsrs D I7=$L l # #Y .E ,^,, #>#>#>#>,1),<,<#>#>#>?!1.!N!N!N ,^,,y,sB!;T!B)BB(7## +gq!,,VI% y!! > >A GCO4JqMK 1 r1==JqMM  J!@R!@!@AAAA'8qAAA nj 88r(ct|dkr|d|d g}tj|dd}fd}t|||||S)z> Calculate null distribution for paired-sample tests. rrr)c`tj|dd}dkr |dd}|d|iS)Nrr)rr)r,r)rr/rr1s r&statistic_wrappedz2_calculate_null_samples..statistic_wrappedsA{4B'' >>!9Dy$*T***r()rr,rr)r/r1rrrDrOrrs ` @r&_calculate_null_samplesrs D I A~~Q$q'" ;tQ # #D++++++ $D*;^$)3 0 00r(c *t|} || krtdhd} |}|| vrtd| d|dvrtd|dtj|jv}|st |}d } t|d kr|d krt| n#t$rt| wxYwt||}g} |D]a} tj | } | j |d krtd tj | | d} | | btj|st|n tj}||ks|dkrtd||}n*t|}||ks|dkrtdhd}|}||vrtd|t#|}| ||||||| |f S)z(Input validation for `permutation_test`.r>rpairings independentz`permutation_type` must be in .rrNrz6`data` must be a tuple containing at least two samplesrRrrrr)rrr>rrrr)rrrrrrr8rrrr,rr#r4risinfrr )r/r1permutation_typerrNrDrrrOrpermutation_typesrrr%rrrs r&_permutation_test_ivr#s4yyH x5666>>>'--//000N:KNNNOOO,,,KLLLw0;;FF 4(33 FG! t99q==->>W%% % !!!   ! T4 ( (DGv&& <  " ":;; ;VXr22v/1x /D/D#s;'''Fo%%A)=)=DEEE }u:: H  A JKK K333L##%%K,&&C\CCDDD S ! !C Y 0*o k8S 22s (CCr)rrrNrDrrrOc *t|||||||| } | \ }}}}}}}}||ddi} tttd} ||||f} | |} | | \}}|rdndt j| jtjsdn t j| jj dz}t j || zfdfdfd }|d }|||| }t j |dd}t| ||S) aQ Performs a permutation test of a given statistic on provided data. For independent sample statistics, the null hypothesis is that the data are randomly sampled from the same distribution. For paired sample statistics, two null hypothesis can be tested: that the data are paired at random or that the data are assigned to samples at random. Parameters ---------- data : iterable of array-like Contains the samples, each of which is an array of observations. Dimensions of sample arrays must be compatible for broadcasting except along `axis`. statistic : callable Statistic for which the p-value of the hypothesis test is to be calculated. `statistic` must be a callable that accepts samples as separate arguments (e.g. ``statistic(*data)``) and returns the resulting statistic. If `vectorized` is set ``True``, `statistic` must also accept a keyword argument `axis` and be vectorized to compute the statistic along the provided `axis` of the sample arrays. permutation_type : {'independent', 'samples', 'pairings'}, optional The type of permutations to be performed, in accordance with the null hypothesis. The first two permutation types are for paired sample statistics, in which all samples contain the same number of observations and observations with corresponding indices along `axis` are considered to be paired; the third is for independent sample statistics. - ``'samples'`` : observations are assigned to different samples but remain paired with the same observations from other samples. This permutation type is appropriate for paired sample hypothesis tests such as the Wilcoxon signed-rank test and the paired t-test. - ``'pairings'`` : observations are paired with different observations, but they remain within the same sample. This permutation type is appropriate for association/correlation tests with statistics such as Spearman's :math:`\rho`, Kendall's :math:`\tau`, and Pearson's :math:`r`. - ``'independent'`` (default) : observations are assigned to different samples. Samples may contain different numbers of observations. This permutation type is appropriate for independent sample hypothesis tests such as the Mann-Whitney :math:`U` test and the independent sample t-test. Please see the Notes section below for more detailed descriptions of the permutation types. vectorized : bool, optional If `vectorized` is set ``False``, `statistic` will not be passed keyword argument `axis` and is expected to calculate the statistic only for 1D samples. If ``True``, `statistic` will be passed keyword argument `axis` and is expected to calculate the statistic along `axis` when passed an ND sample array. If ``None`` (default), `vectorized` will be set ``True`` if ``axis`` is a parameter of `statistic`. Use of a vectorized statistic typically reduces computation time. n_resamples : int or np.inf, default: 9999 Number of random permutations (resamples) used to approximate the null distribution. If greater than or equal to the number of distinct permutations, the exact null distribution will be computed. Note that the number of distinct permutations grows very rapidly with the sizes of samples, so exact tests are feasible only for very small data sets. batch : int, optional The number of permutations to process in each call to `statistic`. Memory usage is O( `batch` * ``n`` ), where ``n`` is the total size of all samples, regardless of the value of `vectorized`. Default is ``None``, in which case ``batch`` is the number of permutations. alternative : {'two-sided', 'less', 'greater'}, optional The alternative hypothesis for which the p-value is calculated. For each alternative, the p-value is defined for exact tests as follows. - ``'greater'`` : the percentage of the null distribution that is greater than or equal to the observed value of the test statistic. - ``'less'`` : the percentage of the null distribution that is less than or equal to the observed value of the test statistic. - ``'two-sided'`` (default) : twice the smaller of the p-values above. Note that p-values for randomized tests are calculated according to the conservative (over-estimated) approximation suggested in [2]_ and [3]_ rather than the unbiased estimator suggested in [4]_. That is, when calculating the proportion of the randomized null distribution that is as extreme as the observed value of the test statistic, the values in the numerator and denominator are both increased by one. An interpretation of this adjustment is that the observed value of the test statistic is always included as an element of the randomized null distribution. The convention used for two-sided p-values is not universal; the observed test statistic and null distribution are returned in case a different definition is preferred. axis : int, default: 0 The axis of the (broadcasted) samples over which to calculate the statistic. If samples have a different number of dimensions, singleton dimensions are prepended to samples with fewer dimensions before `axis` is considered. rng : `numpy.random.Generator`, optional Pseudorandom number generator state. When `rng` is None, a new `numpy.random.Generator` is created using entropy from the operating system. Types other than `numpy.random.Generator` are passed to `numpy.random.default_rng` to instantiate a ``Generator``. Returns ------- res : PermutationTestResult An object with attributes: statistic : float or ndarray The observed test statistic of the data. pvalue : float or ndarray The p-value for the given alternative. null_distribution : ndarray The values of the test statistic generated under the null hypothesis. Notes ----- The three types of permutation tests supported by this function are described below. **Unpaired statistics** (``permutation_type='independent'``): The null hypothesis associated with this permutation type is that all observations are sampled from the same underlying distribution and that they have been assigned to one of the samples at random. Suppose ``data`` contains two samples; e.g. ``a, b = data``. When ``1 < n_resamples < binom(n, k)``, where * ``k`` is the number of observations in ``a``, * ``n`` is the total number of observations in ``a`` and ``b``, and * ``binom(n, k)`` is the binomial coefficient (``n`` choose ``k``), the data are pooled (concatenated), randomly assigned to either the first or second sample, and the statistic is calculated. This process is performed repeatedly, `permutation` times, generating a distribution of the statistic under the null hypothesis. The statistic of the original data is compared to this distribution to determine the p-value. When ``n_resamples >= binom(n, k)``, an exact test is performed: the data are *partitioned* between the samples in each distinct way exactly once, and the exact null distribution is formed. Note that for a given partitioning of the data between the samples, only one ordering/permutation of the data *within* each sample is considered. For statistics that do not depend on the order of the data within samples, this dramatically reduces computational cost without affecting the shape of the null distribution (because the frequency/count of each value is affected by the same factor). For ``a = [a1, a2, a3, a4]`` and ``b = [b1, b2, b3]``, an example of this permutation type is ``x = [b3, a1, a2, b2]`` and ``y = [a4, b1, a3]``. Because only one ordering/permutation of the data *within* each sample is considered in an exact test, a resampling like ``x = [b3, a1, b2, a2]`` and ``y = [a4, a3, b1]`` would *not* be considered distinct from the example above. ``permutation_type='independent'`` does not support one-sample statistics, but it can be applied to statistics with more than two samples. In this case, if ``n`` is an array of the number of observations within each sample, the number of distinct partitions is:: np.prod([binom(sum(n[i:]), sum(n[i+1:])) for i in range(len(n)-1)]) **Paired statistics, permute pairings** (``permutation_type='pairings'``): The null hypothesis associated with this permutation type is that observations within each sample are drawn from the same underlying distribution and that pairings with elements of other samples are assigned at random. Suppose ``data`` contains only one sample; e.g. ``a, = data``, and we wish to consider all possible pairings of elements of ``a`` with elements of a second sample, ``b``. Let ``n`` be the number of observations in ``a``, which must also equal the number of observations in ``b``. When ``1 < n_resamples < factorial(n)``, the elements of ``a`` are randomly permuted. The user-supplied statistic accepts one data argument, say ``a_perm``, and calculates the statistic considering ``a_perm`` and ``b``. This process is performed repeatedly, `permutation` times, generating a distribution of the statistic under the null hypothesis. The statistic of the original data is compared to this distribution to determine the p-value. When ``n_resamples >= factorial(n)``, an exact test is performed: ``a`` is permuted in each distinct way exactly once. Therefore, the `statistic` is computed for each unique pairing of samples between ``a`` and ``b`` exactly once. For ``a = [a1, a2, a3]`` and ``b = [b1, b2, b3]``, an example of this permutation type is ``a_perm = [a3, a1, a2]`` while ``b`` is left in its original order. ``permutation_type='pairings'`` supports ``data`` containing any number of samples, each of which must contain the same number of observations. All samples provided in ``data`` are permuted *independently*. Therefore, if ``m`` is the number of samples and ``n`` is the number of observations within each sample, then the number of permutations in an exact test is:: factorial(n)**m Note that if a two-sample statistic, for example, does not inherently depend on the order in which observations are provided - only on the *pairings* of observations - then only one of the two samples should be provided in ``data``. This dramatically reduces computational cost without affecting the shape of the null distribution (because the frequency/count of each value is affected by the same factor). **Paired statistics, permute samples** (``permutation_type='samples'``): The null hypothesis associated with this permutation type is that observations within each pair are drawn from the same underlying distribution and that the sample to which they are assigned is random. Suppose ``data`` contains two samples; e.g. ``a, b = data``. Let ``n`` be the number of observations in ``a``, which must also equal the number of observations in ``b``. When ``1 < n_resamples < 2**n``, the elements of ``a`` are ``b`` are randomly swapped between samples (maintaining their pairings) and the statistic is calculated. This process is performed repeatedly, `permutation` times, generating a distribution of the statistic under the null hypothesis. The statistic of the original data is compared to this distribution to determine the p-value. When ``n_resamples >= 2**n``, an exact test is performed: the observations are assigned to the two samples in each distinct way (while maintaining pairings) exactly once. For ``a = [a1, a2, a3]`` and ``b = [b1, b2, b3]``, an example of this permutation type is ``x = [b1, a2, b3]`` and ``y = [a1, b2, a3]``. ``permutation_type='samples'`` supports ``data`` containing any number of samples, each of which must contain the same number of observations. If ``data`` contains more than one sample, paired observations within ``data`` are exchanged between samples *independently*. Therefore, if ``m`` is the number of samples and ``n`` is the number of observations within each sample, then the number of permutations in an exact test is:: factorial(m)**n Several paired-sample statistical tests, such as the Wilcoxon signed rank test and paired-sample t-test, can be performed considering only the *difference* between two paired elements. Accordingly, if ``data`` contains only one sample, then the null distribution is formed by independently changing the *sign* of each observation. .. warning:: The p-value is calculated by counting the elements of the null distribution that are as extreme or more extreme than the observed value of the statistic. Due to the use of finite precision arithmetic, some statistic functions return numerically distinct values when the theoretical values would be exactly equal. In some cases, this could lead to a large error in the calculated p-value. `permutation_test` guards against this by considering elements in the null distribution that are "close" (within a relative tolerance of 100 times the floating point epsilon of inexact dtypes) to the observed value of the test statistic as equal to the observed value of the test statistic. However, the user is advised to inspect the null distribution to assess whether this method of comparison is appropriate, and if not, calculate the p-value manually. See example below. References ---------- .. [1] R. A. Fisher. The Design of Experiments, 6th Ed (1951). .. [2] B. Phipson and G. K. Smyth. "Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn." Statistical Applications in Genetics and Molecular Biology 9.1 (2010). .. [3] M. D. Ernst. "Permutation Methods: A Basis for Exact Inference". Statistical Science (2004). .. [4] B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap (1993). Examples -------- Suppose we wish to test whether two samples are drawn from the same distribution. Assume that the underlying distributions are unknown to us, and that before observing the data, we hypothesized that the mean of the first sample would be less than that of the second sample. We decide that we will use the difference between the sample means as a test statistic, and we will consider a p-value of 0.05 to be statistically significant. For efficiency, we write the function defining the test statistic in a vectorized fashion: the samples ``x`` and ``y`` can be ND arrays, and the statistic will be calculated for each axis-slice along `axis`. >>> import numpy as np >>> def statistic(x, y, axis): ... return np.mean(x, axis=axis) - np.mean(y, axis=axis) After collecting our data, we calculate the observed value of the test statistic. >>> from scipy.stats import norm >>> rng = np.random.default_rng() >>> x = norm.rvs(size=5, random_state=rng) >>> y = norm.rvs(size=6, loc = 3, random_state=rng) >>> statistic(x, y, 0) -3.5411688580987266 Indeed, the test statistic is negative, suggesting that the true mean of the distribution underlying ``x`` is less than that of the distribution underlying ``y``. To determine the probability of this occurring by chance if the two samples were drawn from the same distribution, we perform a permutation test. >>> from scipy.stats import permutation_test >>> # because our statistic is vectorized, we pass `vectorized=True` >>> # `n_resamples=np.inf` indicates that an exact test is to be performed >>> res = permutation_test((x, y), statistic, vectorized=True, ... n_resamples=np.inf, alternative='less') >>> print(res.statistic) -3.5411688580987266 >>> print(res.pvalue) 0.004329004329004329 The probability of obtaining a test statistic less than or equal to the observed value under the null hypothesis is 0.4329%. This is less than our chosen threshold of 5%, so we consider this to be significant evidence against the null hypothesis in favor of the alternative. Because the size of the samples above was small, `permutation_test` could perform an exact test. For larger samples, we resort to a randomized permutation test. >>> x = norm.rvs(size=100, random_state=rng) >>> y = norm.rvs(size=120, loc=0.2, random_state=rng) >>> res = permutation_test((x, y), statistic, n_resamples=9999, ... vectorized=True, alternative='less', ... rng=rng) >>> print(res.statistic) -0.4230459671240913 >>> print(res.pvalue) 0.0015 The approximate probability of obtaining a test statistic less than or equal to the observed value under the null hypothesis is 0.0225%. This is again less than our chosen threshold of 5%, so again we have significant evidence to reject the null hypothesis in favor of the alternative. For large samples and number of permutations, the result is comparable to that of the corresponding asymptotic test, the independent sample t-test. >>> from scipy.stats import ttest_ind >>> res_asymptotic = ttest_ind(x, y, alternative='less') >>> print(res_asymptotic.pvalue) 0.0014669545224902675 The permutation distribution of the test statistic is provided for further investigation. >>> import matplotlib.pyplot as plt >>> plt.hist(res.null_distribution, bins=50) >>> plt.title("Permutation distribution of test statistic") >>> plt.xlabel("Value of Statistic") >>> plt.ylabel("Frequency") >>> plt.show() Inspection of the null distribution is essential if the statistic suffers from inaccuracy due to limited machine precision. Consider the following case: >>> from scipy.stats import pearsonr >>> x = [1, 2, 4, 3] >>> y = [2, 4, 6, 8] >>> def statistic(x, y, axis=-1): ... return pearsonr(x, y, axis=axis).statistic >>> res = permutation_test((x, y), statistic, vectorized=True, ... permutation_type='pairings', ... alternative='greater') >>> r, pvalue, null = res.statistic, res.pvalue, res.null_distribution In this case, some elements of the null distribution differ from the observed value of the correlation coefficient ``r`` due to numerical noise. We manually inspect the elements of the null distribution that are nearly the same as the observed value of the test statistic. >>> r 0.7999999999999999 >>> unique = np.unique(null) >>> unique array([-1. , -1. , -0.8, -0.8, -0.8, -0.6, -0.4, -0.4, -0.2, -0.2, -0.2, 0. , 0.2, 0.2, 0.2, 0.4, 0.4, 0.6, 0.8, 0.8, 0.8, 1. , 1. ]) # may vary >>> unique[np.isclose(r, unique)].tolist() [0.7999999999999998, 0.7999999999999999, 0.8] # may vary If `permutation_test` were to perform the comparison naively, the elements of the null distribution with value ``0.7999999999999998`` would not be considered as extreme or more extreme as the observed value of the statistic, so the calculated p-value would be too small. >>> incorrect_pvalue = np.count_nonzero(null >= r) / len(null) >>> incorrect_pvalue 0.14583333333333334 # may vary Instead, `permutation_test` treats elements of the null distribution that are within ``max(1e-14, abs(r)*1e-14)`` of the observed value of the statistic ``r`` to be equal to ``r``. >>> correct_pvalue = np.count_nonzero(null >= r - 1e-14) / len(null) >>> correct_pvalue 0.16666666666666666 >>> res.pvalue == correct_pvalue True This method of comparison is expected to be accurate in most practical situations, but the user is advised to assess this by inspecting the elements of the null distribution that are close to the observed value of the statistic. Also, consider the use of statistics that can be calculated using exact arithmetic (e.g. integer statistics). rr))rrrrrrcX||zk}|dzzz }|SNrrr~rr rr adjustmentrrNs r&rzpermutation_test..less!9 Hu$44888##j0[:5MNr(cX||z k}|dzzz }|Srr~rs r&rz!permutation_test..greater&rr(ch||}||}tj||dz}|Sr)r,r)rr rrrrrs r&rz#permutation_test..two_sided+sBt-x88 !'"3X>>*\?;;a?r(r) rrrrr,r4r;inexactrrrrrS)r/r1rrrNrDrrrOrr null_calculatorsnull_calculator_argscalculate_nullrrrrr!rrrrrs ` @@@@r&rrasL i1A: +UK # % %D $T9& K$y$(R((H$<#:';==!)[!3(%&67N,-/{J!'aJ]8>2:>> 2AA(.))-c1 F3> " "E   !%''G#gk"#4h??Gggq!$$G 74E F FFr(c0eZdZUdZdZeed<dZeed<dS)ResamplingMethodaConfiguration information for a statistical resampling method. Instances of this class can be passed into the `method` parameter of some hypothesis test functions to perform a resampling or Monte Carlo version of the hypothesis test. Attributes ---------- n_resamples : int The number of resamples to perform or Monte Carlo samples to draw. batch : int, optional The number of resamples to process in each vectorized call to the statistic. Batch sizes >>1 tend to be faster when the statistic is vectorized, but memory usage scales linearly with the batch size. Default is ``None``, which processes all resamples in a single batch. rrNNrD)rrrrrNrrrDr!r(r&rr;s>"KE3r(rc>eZdZUdZdZeed<dZeed<ddZdZ dS) MonteCarloMethodaConfiguration information for a Monte Carlo hypothesis test. Instances of this class can be passed into the `method` parameter of some hypothesis test functions to perform a Monte Carlo version of the hypothesis tests. Attributes ---------- n_resamples : int, optional The number of Monte Carlo samples to draw. Default is 9999. batch : int, optional The number of Monte Carlo samples to process in each vectorized call to the statistic. Batch sizes >>1 tend to be faster when the statistic is vectorized, but memory usage scales linearly with the batch size. Default is ``None``, which processes all samples in a single batch. rvs : callable or tuple of callables, optional A callable or sequence of callables that generates random variates under the null hypothesis. Each element of `rvs` must be a callable that accepts keyword argument ``size`` (e.g. ``rvs(size=(m, n))``) and returns an N-d array sample of that shape. If `rvs` is a sequence, the number of callables in `rvs` must match the number of samples passed to the hypothesis test in which the `MonteCarloMethod` is used. Default is ``None``, in which case the hypothesis test function chooses values to match the standard version of the hypothesis test. For example, the null hypothesis of `scipy.stats.pearsonr` is typically that the samples are drawn from the standard normal distribution, so ``rvs = (rng.normal, rng.normal)`` where ``rng = np.random.default_rng()``. rng : `numpy.random.Generator`, optional Pseudorandom number generator state. When `rng` is None, a new `numpy.random.Generator` is created using entropy from the operating system. Types other than `numpy.random.Generator` are passed to `numpy.random.default_rng` to instantiate a ``Generator``. NrrOrch||d}t|||_||_||_||_dS)Nz.Use of `rvs` and `rng` are mutually exclusive.)rrNrDrrO)selfrNrDrrOrs r&__init__zMonteCarloMethod.__init__zs? O#/FGW%% %& r(cPt|j|j|j|jS)N)rNrDrrO)r7rNrDrrOrs r&_asdictzMonteCarloMethod._asdicts+ 0 dh000 0r()rNNN) rrrrrobjectrrOrrr!r(r&rrRsf""FCC00000r(raUse of attribute `random_state` is deprecated and replaced by `rng`. Support for `random_state` will be removed in SciPy 1.19.0. To silence this warning and ensure consistent behavior in SciPy 1.19.0, control the RNG using attribute `rng`. Values set using attribute `rng` will be validated by `np.random.default_rng`, so the behavior corresponding with a given value may change compared to use of `random_state`. For example, 1) `None` will result in unpredictable random numbers, 2) an integer will result in a different stream of random numbers, (with the same distribution), and 3) `np.random` or `RandomState` instances will result in an error. See the documentation of `default_rng` for more information.ceZdZUdZeed<edddZeed<edZ e j dZ ed Z ddd fd Z d Z xZS)PermutationMethodaS Configuration information for a permutation hypothesis test. Instances of this class can be passed into the `method` parameter of some hypothesis test functions to perform a permutation version of the hypothesis tests. Attributes ---------- n_resamples : int, optional The number of resamples to perform. Default is 9999. batch : int, optional The number of resamples to process in each vectorized call to the statistic. Batch sizes >>1 tend to be faster when the statistic is vectorized, but memory usage scales linearly with the batch size. Default is ``None``, which processes all resamples in a single batch. rng : `numpy.random.Generator`, optional Pseudorandom number generator used to perform resampling. If `rng` is passed by keyword to the initializer or the `rng` attribute is used directly, types other than `numpy.random.Generator` are passed to `numpy.random.default_rng` to instantiate a ``Generator`` before use. If `rng` is already a ``Generator`` instance, then the provided instance is used. Specify `rng` for repeatable behavior. If this argument is passed by position, if `random_state` is passed by keyword into the initializer, or if the `random_state` attribute is used directly, legacy behavior for `random_state` applies: - If `random_state` is None (or `numpy.random`), the `numpy.random.RandomState` singleton is used. - If `random_state` is an int, a new ``RandomState`` instance is used, seeded with `random_state`. - If `random_state` is already a ``Generator`` or ``RandomState`` instance then that instance is used. .. versionchanged:: 1.15.0 As part of the `SPEC-007 `_ transition from use of `numpy.random.RandomState` to `numpy.random.Generator`, this attribute name was changed from `random_state` to `rng`. For an interim period, both names will continue to work, although only one may be specified at a time. After the interim period, uses of `random_state` will emit warnings. The behavior of both `random_state` and `rng` are outlined above, but only `rng` should be used in new code. rOFNinitreprdefault_rngc|jSr+ _random_staters r&rzPermutationMethod.random_state !!r(c||_dSr+rrr+s r&rzPermutationMethod.random_state!r(c|jSr+rrs r&rOzPermutationMethod.rng yr(rrOcj||_||_t||dSNrNrD)rrsuperr)rrNrDrrO __class__s r&rzPermutationMethod.__init__s7 ) [>>>>>r(ct|j|j}|j |j|d<|j |j|d<|S)NrrOr)r7rNrDrOrrr\s r&rzPermutationMethod._asdictsG T-TZ @ @ @ 8 xAeH   ( $ 1An r()rNN)rrrrrrrrpropertyrsetterrOrr __classcell__rs@r&rrs..^ KKK5e%>>>D&>>> ""X" !!! X?t???????r(rceZdZUdZeed<edddZeed<dZe ed<e d Z e j d Z e d Z ddd fd ZdZxZS)BootstrapMethoda9 Configuration information for a bootstrap confidence interval. Instances of this class can be passed into the `method` parameter of some confidence interval methods to generate a bootstrap confidence interval. Attributes ---------- n_resamples : int, optional The number of resamples to perform. Default is 9999. batch : int, optional The number of resamples to process in each vectorized call to the statistic. Batch sizes >>1 tend to be faster when the statistic is vectorized, but memory usage scales linearly with the batch size. Default is ``None``, which processes all resamples in a single batch. rng : `numpy.random.Generator`, optional Pseudorandom number generator used to perform resampling. If `rng` is passed by keyword to the initializer or the `rng` attribute is used directly, types other than `numpy.random.Generator` are passed to `numpy.random.default_rng` to instantiate a ``Generator`` before use. If `rng` is already a ``Generator`` instance, then the provided instance is used. Specify `rng` for repeatable behavior. If this argument is passed by position, if `random_state` is passed by keyword into the initializer, or if the `random_state` attribute is used directly, legacy behavior for `random_state` applies: - If `random_state` is None (or `numpy.random`), the `numpy.random.RandomState` singleton is used. - If `random_state` is an int, a new ``RandomState`` instance is used, seeded with `random_state`. - If `random_state` is already a ``Generator`` or ``RandomState`` instance then that instance is used. .. versionchanged:: 1.15.0 As part of the `SPEC-007 `_ transition from use of `numpy.random.RandomState` to `numpy.random.Generator`, this attribute name was changed from `random_state` to `rng`. For an interim period, both names will continue to work, although only one may be specified at a time. After the interim period, uses of `random_state` will emit warnings. The behavior of both `random_state` and `rng` are outlined above, but only `rng` should be used in new code. method : {'BCa', 'percentile', 'basic'} Whether to use the 'percentile' bootstrap ('percentile'), the 'basic' (AKA 'reverse') bootstrap ('basic'), or the bias-corrected and accelerated bootstrap ('BCa', default). rOFNrrrrc|jSr+rrs r&rzBootstrapMethod.random_state' rr(c||_dSr+rrs r&rzBootstrapMethod.random_state- rr(c|jSr+rrs r&rOzBootstrapMethod.rng3 rr(rrcx||_||_||_t||dSr)rrrrr)rrNrDrrrOrs r&rzBootstrapMethod.__init__7 s>  )  [>>>>>r(ct|j|j|j}|j |j|d<|j |j|d<|S)N)rNrDrrOr)r7rNrDrrOrrs r&rzBootstrapMethod._asdictA sR T-TZ  % % % 8 xAeH   ( $ 1An r()rNNr)rrrrrrrrrstrrrrrOrrrrs@r&rrs22f KKK5e%>>>D&>>>FC ""X" !!! XCG?&*???????r(rr+)NN)Ar`numpyr, itertoolsrrrcollections.abcr dataclassesrrrscipy._lib._utilr r r r scipy._lib._array_apir rr scipy.specialrrrr_commonr_axis_nan_policyrr_warnings_errorsr__all__r8rLrPrWrkrrrrrrrr#r1rDr$rSrbrkrwrrrrrrr_rs_deprecationrrr!r(r&rsQ9999999999$$$$$$((((((((222222222222OOOOOOOOOO666666666666''''''GGGGGGGG333333 ? ? ?,(PPP00#0#0#fl+l+l+^ ''''''' ',N##.2$e!d%edBPBPBPBP$#BPJ O8O8O8d """"""" "$8V$$9=!%T{ZFZFZFZF%$ZFz            "F>F>F>R6:d4a6a6a6a6a6H """"""" "$6>>>$$$8"99999999z"&69696969t!%0000>;2;2;2|N##:G $$d!,1$VGVGVGVG$#VGr  , 4040404040'4040 40n " RRRRR(RR Rj ZZZZZ&ZZ ZZZr(