o `^hJ@sdZddlZddlZddlmZddlmZddlmZddl Z ddl Z ddl mZddlmZdd lmZmZdd lmZdd lmZd ed edefddZd ed ede jfddZddZ dddZ dddZ  dddZdS)z9Implementation of ARFF parsers: via LIAC-ARFF and pandas.N) OrderedDict) Generator)List)_arff)ArffSparseDataType)chunk_generatorget_chunk_n_rows)check_pandas_support) pd_fillna arff_datainclude_columnsreturncCstttf}ddt|D}t|d|d|dD] \}}}||vr=|d||d||d||q|S)aObtains several columns from sparse ARFF representation. Additionally, the column indices are re-labelled, given the columns that are not included. (e.g., when including [1, 2, 3], the columns will be relabelled to [0, 1, 2]). Parameters ---------- arff_data : tuple A tuple of three lists of equal size; first list indicating the value, second the x coordinate and the third the y coordinate. include_columns : list A list of columns to include. Returns ------- arff_data_new : tuple Subset of arff data with only the include columns indicated by the include_columns argument. cSi|]\}}||qSr.0 array_idx column_idxrr[/home/air/shanriGPT/back/venv/lib/python3.10/site-packages/sklearn/datasets/_arff_parser.py . z)_split_sparse_columns..rr)list enumeratezipappend)r r arff_data_newreindexed_columnsvalrow_idxcol_idxrrr_split_sparse_columnss"r"c Cs~t|dd}|t|f}ddt|D}tj|tjd}t|d|d|dD]\}}}||vr<|||||f<q+|S)NrcSrrrrrrrr@rz)_sparse_data_to_array..dtyperr)maxlenrnpemptyfloat64r) r r num_obsy_shaperyrr r!rrr_sparse_data_to_array9s "r-cCsP||}t|dkr||}||fSt|dkr"||d}||fSd}||fS)a Post process a dataframe to select the desired columns in `X` and `y`. Parameters ---------- frame : dataframe The dataframe to split into `X` and `y`. feature_names : list of str The list of feature names to populate `X`. target_names : list of str The list of target names to populate `y`. Returns ------- X : dataframe The dataframe containing the features. y : {series, dataframe} or None The series or dataframe containing the target. rrrN)r&)frame feature_names target_namesXr,rrr_post_process_frameKs  r2c"sdd}||}|dkrtjntj}|dk } tj||| d} ||fdd| dD|dkrtd } t| d} t| } t| d }| j |g| d d }|j d d }t |}fdd| D}||g}t | d |D]}|| j || d d |qrt|dkr|d|dj|d<| j|d d}t| |}~~i}|jD]%}|d}|dkrd||<q|dkrd||<q|j|||<q||}t|||\}n| d }fdd|D}fdd|D}t|tr3|durtd|ddkr d}n|d|d}tjtj|d|d }|j|}|dd|f}|dd|fn@t|t rjt!||}t"|dd}|t|f} t#j$j%|d|d|dff| tj&d!}|'}t(||n td"t)|fd#d$|D}!|!snt*|!rt+fd%dt,|Dn t-|!rtd&j.ddkrd'n j.ddkrd|dkr||dfS|dfS)(aARFF parser using the LIAC-ARFF library coded purely in Python. This parser is quite slow but consumes a generator. Currently it is needed to parse sparse datasets. For dense datasets, it is recommended to instead use the pandas-based parser, although it does not always handles the dtypes exactly the same. Parameters ---------- gzip_file : GzipFile instance The file compressed to be read. output_arrays_type : {"numpy", "sparse", "pandas"} The type of the arrays that will be returned. The possibilities ara: - `"numpy"`: both `X` and `y` will be NumPy arrays; - `"sparse"`: `X` will be sparse matrix and `y` will be a NumPy array; - `"pandas"`: `X` will be a pandas DataFrame and `y` will be either a pandas Series or DataFrame. columns_info : dict The information provided by OpenML regarding the columns of the ARFF file. feature_names_to_select : list of str A list of the feature names to be selected. target_names_to_select : list of str A list of the target names to be selected. Returns ------- X : {ndarray, sparse matrix, dataframe} The data matrix. y : {ndarray, dataframe, series} The target. frame : dataframe or None A dataframe containing both `X` and `y`. `None` if `output_array_type != "pandas"`. categories : list of str or None The names of the features that are categorical. `None` if `output_array_type == "pandas"`. css|D]}|dVqdS)Nutf-8)decode) gzip_filelinerrr_io_to_generatorsz+_liac_arff_parser.._io_to_generatorsparsepandas) return_typeencode_nominalcs(i|]\}}t|tr|vr||qSr) isinstancer)rnamecatcolumns_to_selectrrrsz%_liac_arff_parser.. attributeszfetch_openml with as_frame=TruedataF)columnscopyT)deepcg|]}|vr|qSrrrcolr?rr z%_liac_arff_parser..rrr) ignore_index data_typeintegerInt64nominalcategorycg|] }t|dqSindexintrcol_nameopenml_columns_inforrrIcrQrRrTrVrXrrrIrZNz6shape must be provided when arr['data'] is a Generatorr))r$count)shaper$z-Unexpected type for data obtained from arff: csh|]}|vqSrrrV) categoriesrr sz$_liac_arff_parser..c sJg|]!\}}ttj|dddd||dfjtddqS)Or#NrF)rD)r'takeasarraypopastyperU)rirW)r^r,rrrIs zAMix of nominal and non-nominal targets is not currently supported)r[)/rCOO DENSE_GENloadr rrkeysnext DataFrame memory_usagesumr rrr&rddtypesconcatr rClowerr2r<r ValueErrorr'fromiter itertoolschain from_iterablereshapetupler"r%spr8 coo_matrixr)tocsrr-typeallhstackranyr])"r5output_arrays_typerYfeature_names_to_selecttarget_names_to_selectr]r7streamr:r;arff_containerpd columns_info columns_names first_rowfirst_df row_bytes chunksizecolumns_to_keepdfsrBr.rnr= column_dtyper1r feature_indices_to_selecttarget_indices_to_selectr\ arff_data_Xr*X_shapeis_classificationr)r^r@rYr,r_liac_arff_parserks7                                  rc sddl|D]}|ddrnqi|D]}||d}|dkr,d|<q|dkr6d |<qfd d t|D} dd d gd dddd| d } i| |pUi}j|fi|} z dd|D| _Wnty}} zj d| d} ~ ww||fdd| jD} | | } t dfdd}fdd| j D}|D] }| |j|| |<qt| ||\}}|dkr||| dfS||}}fdd | j D}||d|fS)a^ARFF parser using `pandas.read_csv`. This parser uses the metadata fetched directly from OpenML and skips the metadata headers of ARFF file itself. The data is loaded as a CSV file. Parameters ---------- gzip_file : GzipFile instance The GZip compressed file with the ARFF formatted payload. output_arrays_type : {"numpy", "sparse", "pandas"} The type of the arrays that will be returned. The possibilities are: - `"numpy"`: both `X` and `y` will be NumPy arrays; - `"sparse"`: `X` will be sparse matrix and `y` will be a NumPy array; - `"pandas"`: `X` will be a pandas DataFrame and `y` will be either a pandas Series or DataFrame. openml_columns_info : dict The information provided by OpenML regarding the columns of the ARFF file. feature_names_to_select : list of str A list of the feature names to be selected to build `X`. target_names_to_select : list of str A list of the target names to be selected to build `y`. read_csv_kwargs : dict, default=None Keyword arguments to pass to `pandas.read_csv`. It allows to overwrite the default options. Returns ------- X : {ndarray, sparse matrix, dataframe} The data matrix. y : {ndarray, dataframe, series} The target. frame : dataframe or None A dataframe containing both `X` and `y`. `None` if `output_array_type != "pandas"`. categories : list of str or None The names of the features that are categorical. `None` if `output_array_type == "pandas"`. rNr3z@datarLrMrNrOrPcs"i|] \}}|vr||qSrr)rr!r=)rnrrrs z'_pandas_arff_parser..F?%"T\) header index_col na_valueskeep_default_nacomment quotecharskipinitialspace escapecharr$cSsg|]}|qSrr)rr=rrrrIsz'_pandas_arff_parser..zwThe number of columns provided by OpenML does not match the number of columns inferred by pandas when reading the file.crFrrrGr?rrrIrJz^'(?P.*)'$cs"t|}|dur |S|dS)Ncontents)researchgroup) input_stringmatch)single_quote_patternrrstrip_single_quotess  z0_pandas_arff_parser..strip_single_quotescs g|] \}}t|jr|qSr)r<CategoricalDtyperr=r$rrrrIs  r9cs(i|]\}}t|jr||jqSr)r<rr^tolistrrrrrs   )r9r4rp startswithrread_csvrCrqerrors ParserErrorrcompilernitemsr>rename_categoriesr2to_numpy)r5rrYrrread_csv_kwargsr6r=rdtypes_positionaldefault_read_csv_kwargsr.excrrcategorical_columnsrHr1r,r^r)r@rnrrr_pandas_arff_parser7sp8           rcCsD|dkr t||||||S|dkrt||||||Std|d)a6Load a compressed ARFF file using a given parser. Parameters ---------- gzip_file : GzipFile instance The file compressed to be read. parser : {"pandas", "liac-arff"} The parser used to parse the ARFF file. "pandas" is recommended but only supports loading dense datasets. output_type : {"numpy", "sparse", "pandas"} The type of the arrays that will be returned. The possibilities ara: - `"numpy"`: both `X` and `y` will be NumPy arrays; - `"sparse"`: `X` will be sparse matrix and `y` will be a NumPy array; - `"pandas"`: `X` will be a pandas DataFrame and `y` will be either a pandas Series or DataFrame. openml_columns_info : dict The information provided by OpenML regarding the columns of the ARFF file. feature_names_to_select : list of str A list of the feature names to be selected. target_names_to_select : list of str A list of the target names to be selected. read_csv_kwargs : dict, default=None Keyword arguments to pass to `pandas.read_csv`. It allows to overwrite the default options. Returns ------- X : {ndarray, sparse matrix, dataframe} The data matrix. y : {ndarray, dataframe, series} The target. frame : dataframe or None A dataframe containing both `X` and `y`. `None` if `output_array_type != "pandas"`. categories : list of str or None The names of the features that are categorical. `None` if `output_array_type == "pandas"`. z liac-arffr9zUnknown parser: 'z%'. Should be 'liac-arff' or 'pandas'.)rrrq)r5parser output_typerYrrr]rrrrload_arff_from_gzip_files*; r)N)NN)__doc__rsr collectionsrcollections.abcrtypingrnumpyr'scipyrx externalsrexternals._arffrutils._chunkingrr utils._optional_dependenciesr utils.fixesr r"ndarrayr-r2rrrrrrrsJ        # & S