_Mhe(dZddlZddlZddlZddlZddlZddlmZddlZddl Z ddl Z ddl mZmZddlmZmZmZmZmZmZmZmZmZddlmZddlmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.dd l/m0Z0d Z1d Z2d Z3d Z4GddeZ5dZ6GddZ7dZ8eeddZ9eeddZ:eeddZ;eeddZ<GddZ=GddZ>dS)a< Classes for read / write of matlab (TM) 5 files The matfile specification last found here: https://www.mathworks.com/access/helpdesk/help/pdf_doc/matlab/matfile_format.pdf (as of December 5 2008) ================================= Note on functions and mat files ================================= The document above does not give any hints as to the storage of matlab function handles, or anonymous function handles. I had, therefore, to guess the format of matlab arrays of ``mxFUNCTION_CLASS`` and ``mxOPAQUE_CLASS`` by looking at example mat files. ``mxFUNCTION_CLASS`` stores all types of matlab functions. It seems to contain a struct matrix with a set pattern of fields. For anonymous functions, a sub-fields of one of these fields seems to contain the well-named ``mxOPAQUE_CLASS``. This seems to contain: * array flags as for any matlab matrix * 3 int8 strings * a matrix It seems that whenever the mat file contains a ``mxOPAQUE_CLASS`` instance, there is also an un-named matrix (name == '') at the end of the mat file. I'll call this the ``__function_workspace__`` matrix. When I saved two anonymous functions in a mat file, or appended another anonymous function to the mat file, there was still only one ``__function_workspace__`` un-named matrix at the end, but larger than that for a mat file with a single anonymous function, suggesting that the workspaces for the two functions had been merged. The ``__function_workspace__`` matrix appears to be of double class (``mxCLASS_DOUBLE``), but stored as uint8, the memory for which is in the format of a mini .mat file, without the first 124 bytes of the file header (the description and the subsystem_offset), but with the version U2 bytes, and the S2 endian test bytes. There follow 4 zero bytes, presumably for 8 byte padding, and then a series of ``miMATRIX`` entries, as in a standard mat file. The ``miMATRIX`` entries appear to be series of un-named (name == '') matrices, and may also contain arrays of this same mini-mat format. I guess that: * saving an anonymous function back to a mat file will need the associated ``__function_workspace__`` matrix saved as well for the anonymous function to work correctly. * appending to a mat file that has a ``__function_workspace__`` would involve first pulling off this workspace, appending, checking whether there were any more anonymous functions appended, and then somehow merging the relevant workspaces, and saving at the end of the mat file. The mat files I was playing with are in ``tests/data``: * sqr.mat * parabola.mat * some_functions.mat See ``tests/test_mio.py:test_mio_funcs.py`` for the debugging script I was working with. Small fragments of current code adapted from matfile.py by Heiko Henkelmann; parts of the code for simplify_cells=True adapted from http://blog.nephics.com/2019/08/28/better-loadmat-for-scipy/. N)BytesIO) native_code swapped_code) MatFileReader docfillermatdims read_dtype arr_to_charsarr_dtype_number MatWriteError MatReadErrorMatReadWarning) VarReader5) MatlabObjectMatlabFunctionMDTYPES NP_TO_MTYPES NP_TO_MXTYPES miCOMPRESSEDmiMATRIXmiINT8miUTF8miUINT32 mxCELL_CLASSmxSTRUCT_CLASSmxOBJECT_CLASS mxCHAR_CLASSmxSPARSE_CLASSmxDOUBLE_CLASS mclass_info mat_struct)ZlibInputStreamct|tjo0|jdko%|jdkot|dt S)zBDetermine if elem is an array and if first array item is a struct.r) isinstancenpndarraysizendimr")elems U/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/scipy/io/matlab/_mio5.py _has_structr,isF tRZ ( ( ,di!m ,$)a- , tAw + +-cg}|D]}t|tr#|t|:t |r#|t |l|||S)zyConstruct lists from cell arrays (loaded as numpy ndarrays), recursing into items if they contain mat_struct objects.)r%r"append_matstruct_to_dictr,_inspect_cell_array)r' elem_listsub_elems r+r1r1osI'' h + + '   /99 : : : :  " " '   0:: ; ; ; ;   X & & & & r-ci}|jD]^}|j|}t|trt |||<7t |rt |||<Y|||<_|S)z/Construct nested dicts from mat_struct objects.) _fieldnames__dict__r%r"r0r,r1)matobjdfr*s r+r0r0}s} A  q! dJ ' ' %d++AaDD    &t,,AaDDAaDD Hr-c|D]c}t||trt||||<6t||rt ||||<d|S)z,Convert mat objects in dict to nested dicts.)r%r"r0r,r1)r8keys r+_simplify_cellsr<sm11 afj ) ) 1'#//AcFF 3  1(300AcF Hr-cteZdZdZe d fd ZdZdZdZd Z dd Z dd Z d Z xZ S)MatFile5Readera Reader for Mat 5 mat files Adds the following attribute to base class uint16_codec - char codec to use for uint16 char arrays (defaults to system default codec) Uses variable reader that has the following standard interface (see abstract class in ``miobase``:: __init__(self, file_reader) read_header(self) array_from_header(self) and added interface:: set_stream(self, stream) read_full_tag(self) NFTc t||||||||| | stj} | |_d|_d|_dS)zInitializer for matlab 5 file format reader %(matstream_arg)s %(load_args)s %(struct_arg)s uint16_codec : {None, string} Set codec to use for uint16 char arrays (e.g., 'utf-8'). Use system default codec if None N)super__init__sysgetdefaultencoding uint16_codec _file_reader_matrix_reader) self mat_stream byte_order mat_dtype squeeze_mechars_as_stringsmatlab_compatiblestruct_as_record verify_compressed_data_integrityrDsimplify_cells __class__s r+rAzMatFile5Reader.__init__sr*         ,     4133L( "r-c|jd|jd}|jd|dkrdpdS)z3 Guess byte order. Sets stream pointer to 0~rsIM<>)rHseekread)rGmis r+guess_byte_orderzMatFile5Reader.guess_byte_ordersY S!!! _ ! !! $ $ QU{"s)c)r-ci}t|jdd}t|j|}|dd|d<|ddz }|ddz}d ||fz|d <|S) z Read in mat 5 file header dtypes file_header descriptions __header__versionz%d.%d __version__)rrIr rHitemstrip)rGhdict hdr_dtypehdrv_majorv_minors r+read_file_headerzMatFile5Reader.read_file_headersDO,X6}E )44!-05577==lKKli.A%i.4'&'7);;m r-cVt||_t||_dS)za Run when beginning read of variables Sets up readers from parameters in `self` N)rrErFrGs r+initialize_readzMatFile5Reader.initialize_reads*'t,,(..r-c|j\}}|dkstd|j|z}|t krSt |j|}|j||j }|j\}}n!d}|j|j|tkstd|z|j |}||fS)a Read header, return header, next position Header has to define at least .name and .is_global Parameters ---------- None Returns ------- header : object object that can be passed to self.read_var_array, and that has attributes .name and .is_global next_position : int position in stream of next variable rzDid not read any bytesFz$Expecting miMATRIX type here, got %d) rE read_full_tag ValueErrorrHtellrr#rF set_streamrOr TypeError read_header)rGmdtype byte_countnext_posstreamcheck_stream_limitheaders r+read_var_headerzMatFile5Reader.read_var_headers"".<<>> A~~566 6?''))J6 \ ! !$T_jAAF   * *6 2 2 2!%!F !%!4!B!B!D!D FJJ!&    * *4? ; ; ;!!BVKLL L$001CDDxr-c8|j||S)a Read array, given `header` Parameters ---------- header : header object object with fields defining variable header process : {True, False} bool, optional If True, apply recursive post-processing during loading of array. Returns ------- arr : array array with post-processing applied or not according to `process`. )rFarray_from_header)rGr{processs r+read_var_arrayzMatFile5Reader.read_var_arrays""44VWEEEr-c t|tr|g}n|t|}|jd||}g|d<|sf|\}}|j dn|j d}||vr"d|d}tj |td |d krd }d }nd }|||vr|j| |||}n?#t$r2} tj d|d| dt d d| }Yd} ~ nd} ~ wwxYw|j||||<|jr|d||)||t)|dkrn|f|jrt-|S|S)z get variables from stream as dictionary variable_names - optional list of variable names to get If variable_names is None, then get all variables in file Nr __globals__Nonelatin1zDuplicate variable name "z{" in stream - replacing previous with new Considerscipy.io.matlab.varmats_from_mat to split file into single variable filesrT) stacklevel__function_workspace__FTzUnreadable variable "z ", because ""z Read error: )r%strlistrHrWrnrk end_of_streamr|namedecodewarningswarnrrrWarning is_globalr/removelenrPr<) rGvariable_namesmdictrh next_positionrmsgrreserrs r+ get_variableszMatFile5Reader.get_variables%st nc * * 2,-NN  '!.11N Q %%''!m$$&&$ !%!5!5!7!7 C X-6638??83L3LDu}}6666  c>a@@@@rzz/ )d..H.H$$]333 +))#w77 + + + DDDDcDDD+++++S**  + O  / / /E$K} 2m$++D111)%%d+++~&&!++I$$&&$ J   "5)) )LsD-- E)7(E$$E)cB|jd||g}|s|\}}|jdn|jd}|dkrd}|j |}|j rd}ntj |j d}||||f|j|||S) z list variables from stream rNrrrrlogicalunknown)rHrWrnrkrr|rrrFshape_from_header is_logicalr!getmclassr/)rGvarsrhrrshapeinfos r+list_variableszMatFile5Reader.list_variables`s Q  $$&& 0!%!5!5!7!7 C X-6638??83L3LDrzz/'99#>>E~ > "sz9== KKud+ , , , O  / / /$$&& 0 r-) NFFTFTTNF)TN)__name__ __module__ __qualname____doc__rrArZrkrnr|rrr __classcell__)rQs@r+r>r>s&! !"&#("&26" %$#$#$#$#$#Y$#L***    / / /! ! ! FFFFF&9999vr-r>cRt|}|dttddj}||}|d|||}g}| s|}| \}}|j dn|j d}||||z } || } t} | || | | d||| f| |S)a- Pull variables out of mat 5 file as a sequence of mat file objects This can be useful with a difficult mat file, containing unreadable variables. This routine pulls the variables out in raw form and puts them, unread, back into a file stream for saving or reading. Another use is the pathological case where there is more than one variable of the same name in the file; this routine returns the duplicates, whereas the standard reader will overwrite duplicates in the returned dictionary. The file pointer in `file_obj` will be undefined. File pointers for the returned file-like objects are set at 0. Parameters ---------- file_obj : file-like file object containing mat file Returns ------- named_mats : list list contains tuples of (name, BytesIO) where BytesIO is a file-like object containing mat file contents as for a single variable. The BytesIO contains a string with the original header and a single var. If ``var_file_obj`` is an individual BytesIO instance, then save as a mat file with something like ``open('test.mat', 'wb').write(var_file_obj.read())`` Examples -------- >>> import scipy.io >>> import numpy as np >>> from io import BytesIO >>> from scipy.io.matlab._mio5 import varmats_from_mat >>> mat_fileobj = BytesIO() >>> scipy.io.savemat(mat_fileobj, {'b': np.arange(10), 'a': 'a string'}) >>> varmats = varmats_from_mat(mat_fileobj) >>> sorted([name for name, str_obj in varmats]) ['a', 'b'] rr\r]Nrr)r>rWrritemsizerXrnrkrrrr|rrrwriter/) file_objrdrhdr_lenraw_hdrr named_matsstart_positionrhrrwvar_strout_objs r+varmats_from_matrysP  " "C MM!k"8,];DGmmG$$G MM!MMOOMJ!! +& 0022])vvsxx/H/H n%%%"^3 -- ++)) g g Q4/***!! + r-ceZdZdZdS)EmptyStructMarkerz= Class to indicate presence of empty matlab struct on output N)rrrrr-r+rrsGGGGr-rct|tjr|S|dSt|drtj|St|dot|dot|d}t|tjrn7|s5t|dr%d|jD}d}|rg}g}|D]b\}}t|trH|d d vr>| t|tf| |c|r#tj t|g|StS tj|}n+#t$rtj|t }YnwxYw|jjttjfvr|jd kr||krdS|S) a Convert input object ``source`` to something we can write Parameters ---------- source : object Returns ------- arr : None or ndarray or EmptyStructMarker If `source` cannot be converted to something we can write to a matfile, return None. If `source` is equivalent to an empty dictionary, return ``EmptyStructMarker``. Otherwise return `source` converted to an ndarray with contents for writing to matfile. N __array__keysvaluesitemsr6cDi|]\}}|d||S)_) startswith).0r;values r+ z to_writeable..s>222e ^^C002#u222r-Tr _0123456789dtyper)r%r&r'hasattrasarraygenericr6rrr/objectarraytupler asanyarrayrqrtypeobject_r)source is_mappingrrfieldrnarrs r+ to_writeablers&"*%%  ~tv{##"z&!!!&&))+gfh.G.G+&'**&"*%% GFJ7722v/D/D/F/F222  %"LLNN % %LE55#&& %!HM11 c%jj&1222 e$$$  %8U6]]OU33 3$ $3}V$$ 333}V62223 z62:... zRDFNNt Ks1F%F.-F.r\r]tag_full tag_smalldata array_flagsceZdZdZejdeZeed<dZ dZ dZ ddZ d Z d Z dd ZdZdZdZdZddZdZdZdZdZdZdZdS) VarWriter5z% Generic matlab matrix writing class rrvc|j|_|j|_|j|_|j|_d|_d|_dS)NF) file_streamunicode_stringslong_field_namesoned_as _var_name_var_is_global)rG file_writers r+rAzVarWriter5.__init__sB&2*: + <"* #r-cb|j|ddS)NForder)rrtobytesrGarrs r+ write_byteszVarWriter5.write_bytess- s{{{5566666r-c:|j|dSr)rr)rGss r+ write_stringzVarWriter5.write_strings q!!!!!r-Ncx|t|jjdd}|jjtkr>||j}|j|j z}|dkr| |||dS| |||dS)z write tag and data Nr) rrr byteorderrbyteswapview newbyteorderr(rwrite_smalldata_elementwrite_regular_element)rGrrvrws r+ write_elementzVarWriter5.write_elements >!#)-"34F 9 , . .,,..%%ci&<&<&>&>??CXcl* ??  ( (fj A A A A A  & &sFJ ? ? ? ? ?r-ctjdt}|dz|z|d<|d|d<||dS)Nrbyte_count_mdtyperrdata)r&zeros NDT_TAG_SMALLrr)rGrrvrwtags r+rz"VarWriter5.write_smalldata_element!sYhr=))$."$4#>  kkk,,F  r-ctjdt}||d<||d<|||||dz}|r"|jdd|z zdSdS)Nrrvrwra)r&r NDT_TAG_FULLrrr)rGrrvrwrbc_mod_8s r+rz VarWriter5.write_regular_element)shr<((H &L  >  ;   " "7aj#9 : : : : : ; ;r-Frcb|j}|j}|j|_||jtjdt}t|d<d|d<|dz|dzz|dzz} || dzz|d<||d <||| tj |d tj |}|d kr||td n| |td |_d|_dS)a Write header for given data options shape : sequence array shape mclass - mat5 matrix class is_complex - True if matrix is complex is_logical - True if matrix is logical nzmax - max non zero elements for sparse arrays We get the name and the global flag from the object, and reset them to defaults after we've used them r data_typerarwrTr flags_classnzmaxi4rrrFN)rrrrr _mat_tag_posrmat_tagr&rNDT_ARRAY_FLAGSrrrrrr) rGrr is_complexrrrrafflagss r+ write_headerzVarWriter5.write_header5s0$~'  ,1133 &&& Xb/ * *";<a)q.0:?B"UaZ/=7   28E666777z$ 2::  ( (vq 9 9 9 9   tV , , ,#r-c"|j}|j|||z dz }|dkrtd||jd<||j|j|dS)Nralz-Matrix too large to save with Matlab 5 formatrw)rrrrWr r r)rG start_poscurr_posrws r+update_matrix_tagzVarWriter5.update_matrix_tag`s#((** i((( )A-   !+,, ,%/ \" &&& h'''''r-cL||_||_||dS)a Write variable at top level of mat file Parameters ---------- arr : array_like array-like object to create writer for name : str, optional name as it will appear in matlab workspace default is empty string is_global : {False, True}, optional whether variable will be global on load into matlab N)rrr)rGrrrs r+ write_topzVarWriter5.write_topks&( 3r-cN|j}tj|r,||||dSt|}|#td|dt|dt|tr| |nt|trtd|tur|n|jjr||nh|jjr||nF|jjdvr#|jrd}nd}|||n||||dS) z Write `arr` to stream at top and sub levels Parameters ---------- arr : array_like array-like object to create writer for NzCould not convert z (type z ) to arrayzCannot write matlab functions)USUTF8ascii)rrrscipysparseissparse write_sparserrrtrr%r write_objectrr rwrite_empty_structrfields write_struct hasobject write_cellskindr write_char write_numeric)rGr mat_tag_posrcodecs r+rzVarWriter5.writes&++-- <  % %    c " " "  " "; / / / FC   <RRRT#YYRRRSS S dL ) ) %   d # # # # n - - % ?@@ @ & & &  # # % % % % Z  %   d # # # # Z ! %   T " " " " Z_ * *#  OOD% ( ( ( (   t $ $ $ {+++++r-c2|jjdk}|jjdk} t|jjdd}n\#t$rO|r|d}n-|r|d}n|d}t }YnwxYw|t||j ||||r6| |j | |j dS| |dS)Ncbrc128i1f8)r r) rr%rrKeyErrorastyper rr rrrealimag)rGrimagflogifrs r+r'zVarWriter5.write_numerics5 #% #% $"39=#45FF $ $ $ 'jj(( 'jj&&jj&&#FFF $ '#t|44 %*%*  , , ,  $   sx ( ( (   sx ( ( ( ( (   s # # # # #sAABBrc|jdkstj|dkrWdtj|jdgz}||t ||tddSt|}|j }||t |j j dkr|jrtj|}tjdt!|||j}||}tjt+|fd |}||t dS) z5 Write string array `arr` with given `codec` rrrrTNrrrrbufferS1rv)r(r&allmaxr)rrrrr rrr%mathprodr'r Tcopyrdencoderr)rGrr)rn_charsst_arrsts r+r&zVarWriter5.write_chars[ 8q==BF3"9--=2638Q-000E   e\ 2 2 2  ( (fa 8 8 8 F3  %... 9>S SX i&&GZb&6sG&D&D'*uzz||555F%%e,,B*CGG:#'$&(((C 3v.....r-cP|}||jjdk}|jjdk}|j}|t ||jt|||dkrdn|| |j d| |j d| |j j|r!| |j jdSdS)z Sparse matrices are 2D r+r,rr)r rrrN)tocsc sort_indicesrr%nnzrr rrrindicesr1indptrrr2r3)rGrAr rnzs r+rzVarWriter5.write_sparses IIKK glc) glc) U '#t|44(%/%/%'1WW"  6 6 6 19++D11222 18??400111 16;'''  ,   qv{ + + + + + , ,r-c|t||jtt j|d}|D]}||dS)Nr)rr rrr& atleast_2dflattenr)rGrrLels r+r$zVarWriter5.write_cellsst '#t|44& ( ( ( M#   & &s + +  B JJrNNNN  r-c|dt|tjdtj|tjgtjdS)N)rrrr)rrrr&rint32int8rms r+r zVarWriter5.write_empty_structsh &.111 28ARX666777 28Bbg66677777r-c|t||jt||dSr)rr rr _write_itemsrs r+r"zVarWriter5.write_struct sE '#t|44( * * * #r-cd|jjD}td|Ddz}|jrdpd}||krt d|dz z|t j|gd|t j|d |zt t j | d }|D]"}|D]}| ||#dS) Ncg|] }|d Sr7r)rr9s r+ z+VarWriter5._write_items..s444qad444r-c,g|]}t|Sr)r)r fieldnames r+rYz+VarWriter5._write_items..sAAAc)nnAAAr-r@ z+Field names are restricted to %d charactersrrzS%dr;r) rdescrr=rrqrr&rrrOrPr)rGr fieldnameslength max_lengthrLrQr9s r+rVzVarWriter5._write_itemss944CIO444 AAjAAABB1D+29r J  J(l,-- - 28VHD999:::  HZu'7 8 8 8     M#   & &s + + " "B " " 2a5!!!! " " "r-c|t||jt|t j|jdt| |dS)zmSame as writing structs, except different mx class, and extra classname element after header rrr;N) rr rrrr&r classnamerrVrs r+rzVarWriter5.write_object#sz '#t|44( * * * 28CM==="(  * * * #r-r)FFr)r)rrrrr&rrr rrArrrrrrrrrr'r&rr$r r"rVrrr-r+rrsf//bhr<((G GH$$$777""" @ @ @ @ ; ; ;!& % )$)$)$)$V ( ( ((%,%,%,N$$$2)/)/)/)/V,,,(888 """"r-rcBeZdZdZe ddZdZd dZdS) MatFile5Writerz Class for writing mat5 files FNrowc|||_||_||_|r||_ng|_||_||_d|_dS)a< Initialize writer for matlab 5 format files Parameters ---------- %(do_compression)s %(unicode_strings)s global_vars : None or sequence of strings, optional Names of variables to be marked as global for matlab %(long_fields)s %(oned_as)s N)rdo_compressionr global_varsrr_matrix_writer)rGrrhrrirrs r+rAzMatFile5Writer.__init__1sT$',.  "*D  !D  0 "r-c@tjdt}dtjdt j|d<d|d<tjddtjd |d <|j | dS) NrzMATLAB 5.0 MAT-file Platform: z, Created on: r^r`S2iIMr8 endian_test) r&r NDT_FILE_HDRosrtimeasctimer'uint16rrr)rGrhs r+write_file_headerz MatFile5Writer.write_file_headerNshr<((>rw>>-1\^^>>MIZb,0-/Yv->->@@@M s{{}}-----r-cB||jdk}|r|t||_|D]?\}}|ddkr||jv}|jrt}||j_|j || d|tj | }tjdt }t"|d<t%||d<|j||j||j || d|AdS)a Write variables in `mdict` to stream Parameters ---------- mdict : mapping mapping with method ``items`` returns name, contents pairs where ``name`` which will appear in the matlab workspace in file load, and ``contents`` is something writeable to a matlab file, such as a NumPy array. write_header : {None, True, False}, optional If True, then write the matlab file header before writing the variables. If None (the default) then write the file header if we are at position 0 in the stream. By setting False here, and setting the stream position to the end of the file, you can append variables to a matlab file Nrrrrrvrw)rrrrtrrjrrirhrrrBzlibcompressgetvaluer&emptyrrrrr) rGrrrvarrryout_strrs r+ put_variableszMatFile5Writer.put_variablesYs$  +0022a7L  %  " " $ $ $(.. U UID#Aw#~~ 00I" U 28#/#--c4;;x3H3H)TTT-(9(9::hr<00 ,H $'LLL! &&s{{}}555 &&w////#--c4;;x3H3H)TTTT U Ur-)FFNFrfr)rrrrrrArtr|rr-r+rere.sp(( %!&!"' ###Y#8 . . .&U&U&U&U&U&Ur-re)?rr>rprqrBrviorrnumpyr& scipy.sparser_byteordercodesrr_miobaserrr r r r r rr _mio5_utilsr _mio5_paramsrrrrrrrrrrrrrrrr r!r"_streamsr#r,r1r0r<r>rrrrorrr rrerr-r+rsEEN 666666664444444444444444444444 $#####CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC &%%%%%---          aaaaa]aaaHAAAHHHHHHHHH555r{#H-m< {#H-j9  $X.? +&x0?kkkkkkkk\ QUQUQUQUQUQUQUQUQUQUr-