0PhLQdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z ddl Z ddlZddlmZddlmZddlmZddlmZmZdd lmZmZmZdd lmZd d lm Z m!Z!d d l"m#Z#m$Z$m%Z%m&Z&m'Z'ej(e)Z*e#dddZ+dZ,dZ-dZ.dZ/dZ0ej1dZ2dZ3dZ4ee5ej6dgehdgddgdgdge7gdgdgee d ddgee d dd!gd" d#$dd%dd#d&d'd#d(d)d*d" d+Z8eehdge7ge5ej6dgdgdgdgdgee d ddgee d dd!gd, d#$d%d'dd#d(d#d(d)d*d, d-Z9dS).aCaching loader for the 20 newsgroups text classification dataset. The description of the dataset is available on the official website at: http://people.csail.mit.edu/jrennie/20Newsgroups/ Quoting the introduction: The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his Newsweeder: Learning to filter netnews paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering. This dataset loader will download the recommended "by date" variant of the dataset and which features a point in time split between the train and test sets. The compressed dataset size is around 14 Mb compressed. Once uncompressed the train set is 52 MB and the test set is 34 MB. N)suppress)IntegralReal) preprocessing)CountVectorizer)Bunchcheck_random_state)Interval StrOptionsvalidate_params)tarfile_extractall) get_data_home load_files)RemoteFileMetadata_convert_data_dataframe _fetch_remote _pkl_filepath load_descrz20news-bydate.tar.gzz.https://ndownloader.figshare.com/files/5975967@8f1b2514ca22a5ade8fbb9cfa5727df95fa587f4c87b786e15c759fa66d95610)filenameurlchecksumz20news-bydate.pkzz20news-bydate-trainz20news-bydate-testctj|t}tj|t}tj|dt dtj tt|||}t d|tj |d5}t||dddn #1swxYwYtt 5tj|dddn #1swxYwYt%t'|d t'|d  }t)jt-j|d } t|d 5} | | dddn #1swxYwYt3j||S)zADownload the 20 newsgroups data and stored it as a zipped pickle.T)exist_okz#Downloading dataset from %s (14 MB))dirname n_retriesdelayzDecompressing %szr:gz)pathNlatin1)encodingtraintest zlib_codecwb)osr join TRAIN_FOLDER TEST_FOLDERmakedirsloggerinfoARCHIVErrdebugtarfileopenrrFileNotFoundErrorremovedictrcodecsencodepickledumpswriteshutilrmtree) target_dir cache_pathrr train_path test_path archive_pathfpcachecompressed_contentfs c/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/sklearn/datasets/_twenty_newsgroups.py_download_20newsgroupsrGGsYj,77J Z55IK T**** KK5w{CCC yL LL#\222 lF + +0r2J////000000000000000 # $ $   ,                h777  H 5 5 5   E v|E':':LII j$  $1 "###$$$$$$$$$$$$$$$ M* Ls6C,,C03C0 D,,D03D0F??GGc8|d\}}}|S)z Given text in "news" format, strip the headers, by removing everything before the first blank line. Parameters ---------- text : str The text from which to remove the signature block. z ) partition)text_before _blanklineafters rFstrip_newsgroup_headerrNgs!"&!7!7GZ LzF(writes in|writes:|wrote:|says:|said:|^In article|^Quoted from|^\||^>)cjd|dD}d|S)a: Given text in "news" format, strip lines beginning with the quote characters > or |, plus lines that often introduce a quoted section (for example, because they contain the string 'writes:'.) Parameters ---------- text : str The text from which to remove the signature block. cFg|]}t||S) _QUOTE_REsearch).0lines rF z+strip_newsgroup_quoting..s+RRR49;K;KD;Q;QR$RRRrO )splitr))rJ good_liness rFstrip_newsgroup_quotingr[zs6SR4::d#3#3RRRJ 99Z  rOcL|d}tt|dz ddD]7}||}|ddkrn8|dkrd|d|S|S)a Given text in "news" format, attempt to remove a signature block. As a rough heuristic, we assume that signatures are set apart by either a blank line or a line made of hyphens, and that it is the last such line in the file (disregarding blank lines at the end). Parameters ---------- text : str The text from which to remove the signature block. rXr-rN)striprYrangelenr))rJlinesline_numrVs rFstrip_newsgroup_footerres JJLL  t $ $E#e**q."b11X ::<<  c " "b ( ( E )!||yyyy)*** rO>allr%r$z array-likeboolean random_stateleft)closedgneither) data_homesubset categoriesshufflerhr4download_if_missing return_X_yrrT)prefer_skip_nested_validationr$*rRFg?c vt|}t|t} tj|d} d} tj| r t| d5} | }dddn #1swxYwYtj |d}tj |} nS#t$rF}tdtdtdt|Yd}~nd}~wwxYw| ?|r.tdt#| | || } nt%d |d vr | |n|d krt'}t'}t'}d D]X}| ||j|j|jY|_t1j|_t1j|_t5d }|_d|vrdjD_d|vrdjD_d|vrdjD_|fd|D}|t;|\}}t1jj|}j|_j|_t1j|j_t'|_ t1jjtB}||}|"_|rtG|}t1j$jj%d}|&|j|_j|_t1jjtB}||}|"_|rjjfSS)aLoad the filenames and data from the 20 newsgroups dataset (classification). Download it if necessary. ================= ========== Classes 20 Samples total 18846 Dimensionality 1 Features text ================= ========== Read more in the :ref:`User Guide <20newsgroups_dataset>`. Parameters ---------- data_home : str or path-like, default=None Specify a download and cache folder for the datasets. If None, all scikit-learn data is stored in '~/scikit_learn_data' subfolders. subset : {'train', 'test', 'all'}, default='train' Select the dataset to load: 'train' for the training set, 'test' for the test set, 'all' for both, with shuffled ordering. categories : array-like, dtype=str, default=None If None (default), load all the categories. If not None, list of category names to load (other categories ignored). shuffle : bool, default=True Whether or not to shuffle the data: might be important for models that make the assumption that the samples are independent and identically distributed (i.i.d.), such as stochastic gradient descent. random_state : int, RandomState instance or None, default=42 Determines random number generation for dataset shuffling. Pass an int for reproducible output across multiple function calls. See :term:`Glossary `. remove : tuple, default=() May contain any subset of ('headers', 'footers', 'quotes'). Each of these are kinds of text that will be detected and removed from the newsgroup posts, preventing classifiers from overfitting on metadata. 'headers' removes newsgroup headers, 'footers' removes blocks at the ends of posts that look like signatures, and 'quotes' removes lines that appear to be quoting another post. 'headers' follows an exact standard; the other filters are not always correct. download_if_missing : bool, default=True If False, raise an OSError if the data is not locally available instead of trying to download the data from the source site. return_X_y : bool, default=False If True, returns `(data.data, data.target)` instead of a Bunch object. .. versionadded:: 0.22 n_retries : int, default=3 Number of retries when HTTP errors are encountered. .. versionadded:: 1.5 delay : float, default=1.0 Number of seconds between retries. .. versionadded:: 1.5 Returns ------- bunch : :class:`~sklearn.utils.Bunch` Dictionary-like object, with the following attributes. data : list of shape (n_samples,) The data list to learn. target: ndarray of shape (n_samples,) The target labels. filenames: list of shape (n_samples,) The path to the location of the data. DESCR: str The full description of the dataset. target_names: list of shape (n_classes,) The names of target classes. (data, target) : tuple if `return_X_y=True` A tuple of two ndarrays. The first contains a 2D array of shape (n_samples, n_classes) with each row representing one sample and each column representing the features. The second array of shape (n_samples,) contains the target samples. .. versionadded:: 0.22 Examples -------- >>> from sklearn.datasets import fetch_20newsgroups >>> cats = ['alt.atheism', 'sci.space'] >>> newsgroups_train = fetch_20newsgroups(subset='train', categories=cats) >>> list(newsgroups_train.target_names) ['alt.atheism', 'sci.space'] >>> newsgroups_train.filenames.shape (1073,) >>> newsgroups_train.target.shape (1073,) >>> newsgroups_train.target[:10] array([0, 1, 1, 1, 0, 1, 1, 0, 0, 0]) rl 20news_homeNrbr&P________________________________________________________________________________zCache loading failedz8Downloading 20news dataset. This may take a few minutes.)r=r>rrz20Newsgroups dataset not foundr#rftwenty_newsgroups.rstheadersc,g|]}t|SrR)rNrUrJs rFrWz&fetch_20newsgroups..^!HHHd+D11HHHrOfootersc,g|]}t|SrR)rer}s rFrWz&fetch_20newsgroups..`r~rOquotesc,g|]}t|SrR)r[r}s rFrWz&fetch_20newsgroups..bs!IIIt,T22IIIrOcHg|]}j||fSrR) target_namesindex)rUcatdatas rFrWz&fetch_20newsgroups..es/LLL#4$**3//5LLLrOdtyper)'rr CACHE_NAMEr(r r)existsr2readr6decoder8loads Exceptionprintr-r.rGOSErrorlistextendrtarget filenamesnparrayrDESCRsortzipisin searchsortedrobjecttolistr arangeshapero)rlrmrnrorhr4rprqrrr> twenty_homerCrErDuncompressed_contentedata_lstrrfdescrlabelsmaskindicesrs @rFfetch_20newsgroupsrs>V 222Iy*55J',,y-88K E w~~j!!  j$'' .1%&VVXX" . . . . . . . . . . . . . . .#)=1C\#R#R L!566EE    (OOO ( ) ) ) (OOO !HHHHHHHH    }  < KKR S S S*&%# EE:;; ; """V} 566FF ' - -F=D OODI & & & MM$+ & & &   T^ , , , , hv&& ),, / 0 0FDJFHHdiHHH FHHdiHHH 6IItyIII LLLLLLL  &\ wt{F++-k$' ofdk::  ,,8DIV444D>OO%%  &),77 )DK-a011W%%%0k'* 8DIV444G$OO%% &y$+%% Ks<)C9B CBC!B",C D`. Parameters ---------- subset : {'train', 'test', 'all'}, default='train' Select the dataset to load: 'train' for the training set, 'test' for the test set, 'all' for both, with shuffled ordering. remove : tuple, default=() May contain any subset of ('headers', 'footers', 'quotes'). Each of these are kinds of text that will be detected and removed from the newsgroup posts, preventing classifiers from overfitting on metadata. 'headers' removes newsgroup headers, 'footers' removes blocks at the ends of posts that look like signatures, and 'quotes' removes lines that appear to be quoting another post. data_home : str or path-like, default=None Specify an download and cache folder for the datasets. If None, all scikit-learn data is stored in '~/scikit_learn_data' subfolders. download_if_missing : bool, default=True If False, raise an OSError if the data is not locally available instead of trying to download the data from the source site. return_X_y : bool, default=False If True, returns ``(data.data, data.target)`` instead of a Bunch object. .. versionadded:: 0.20 normalize : bool, default=True If True, normalizes each document's feature vector to unit norm using :func:`sklearn.preprocessing.normalize`. .. versionadded:: 0.22 as_frame : bool, default=False If True, the data is a pandas DataFrame including columns with appropriate dtypes (numeric, string, or categorical). The target is a pandas DataFrame or Series depending on the number of `target_columns`. .. versionadded:: 0.24 n_retries : int, default=3 Number of retries when HTTP errors are encountered. .. versionadded:: 1.5 delay : float, default=1.0 Number of seconds between retries. .. versionadded:: 1.5 Returns ------- bunch : :class:`~sklearn.utils.Bunch` Dictionary-like object, with the following attributes. data: {sparse matrix, dataframe} of shape (n_samples, n_features) The input data matrix. If ``as_frame`` is `True`, ``data`` is a pandas DataFrame with sparse columns. target: {ndarray, series} of shape (n_samples,) The target labels. If ``as_frame`` is `True`, ``target`` is a pandas Series. target_names: list of shape (n_classes,) The names of target classes. DESCR: str The full description of the dataset. frame: dataframe of shape (n_samples, n_features + 1) Only present when `as_frame=True`. Pandas DataFrame with ``data`` and ``target``. .. versionadded:: 0.24 (data, target) : tuple if ``return_X_y`` is True `data` and `target` would be of the format defined in the `Bunch` description above. .. versionadded:: 0.20 Examples -------- >>> from sklearn.datasets import fetch_20newsgroups_vectorized >>> newsgroups_vectorized = fetch_20newsgroups_vectorized(subset='test') >>> newsgroups_vectorized.data.shape (7532, 130107) >>> newsgroups_vectorized.target.shape (7532,) rv20newsgroup_vectorizedzremove-r^z.pklr$NT ) rlrmrnrorhr4rprrr%zThe cached dataset located in z was fetched with an older scikit-learn version and it is not compatible with the scikit-learn version imported. You need to manually delete the file: .r )compressF)copyrfrzcategory_classfetch_20newsgroups_vectorized)r sparse_data)rrframer feature_namesr)rr)rrr(r rjoblibload ValueErrorrrint16 fit_transformrtocsr transformget_feature_names_outdumpastypefloat64rrrrspvstack concatenaterrr )rmr4rlrprqrrrrfilebase target_file data_train data_testX_trainX_testrr vectorizerrrrrr target_names rFrrs1\ 222I'H 1I 0 000 8f+<==K$/   J#/   I w~~k""O -3[-E-E *GV]]   <<<.9<<<    %28444 **:?;;AACC%%in55;;=="88::  Wfm4kANNNN4..,,rz**e4444U3333*L " 6  ! 5y'6*++1133!2I4D EFF / 0 0F E#$K 5 +   $    tVV|  !#    sB,, C6C  C):__doc__r6loggingr(r8rer;r1 contextlibrnumbersrrrnumpyr scipy.sparsesparserr_rfeature_extraction.textrutilsr r utils._param_validationr r r utils.fixesrrr_baserrrrr getLogger__name__r-r/rr*r+rGrNcompilerSr[restrPathLiketuplerrrRrOrFrs6  """""""" 555555--------KKKKKKKKKK,,,,,,''''''''  8 $ $   #8 O    ! $ " @    BJQ   ! ! !22;-:666778#T*;'(' ){ khxD@@@A(4d9===>  #'"    QQQQQh:666778'2;- ){ k[KhxD@@@A(4d9===>  #'       ^^^^  ^^^rO