:py:mod:`pemtk.fit._io`
=======================

.. py:module:: pemtk.fit._io


Module Contents
---------------


Functions
~~~~~~~~~

.. autoapisummary::

   pemtk.fit._io.setTimeStampedFileName
   pemtk.fit._io.writeFitData
   pemtk.fit._io.processedToHDF5
   pemtk.fit._io._pickleData
   pemtk.fit._io._writePDData
   pemtk.fit._io._writeXRData
   pemtk.fit._io.getFilesList
   pemtk.fit._io.loadFitData



.. py:function:: setTimeStampedFileName(self, outStem=None, n=None, ext='pickle', timeString=None, timeFormat='%d%m%y_%H-%M-%S')

   Set unique filename as f'{outStem}_n{n}_{timeString.strftime("%d%m%y_%H-%M-%S")}.{ext}'

   :param outStem: Stem for output file.
                   If None, set to 'PEMtk_data_dump'
   :type outStem: str, optional, default = None
   :param n: Int index to include in file name.
             If None, this will be omitted.
   :type n: int, optional, default = None
   :param ext: File ending.
   :type ext: str, optional, default = 'pickle'
   :param timeString: Timestamp for the file.
                      If None, current time will be used.
   :type timeString: Datatime object, default = None
   :param timeFormat:
   :type timeFormat: Datatime format string, optional, default = "%d%m%y_%H-%M-%S"
   :param TODO:
   :type TODO: additional formatting options, data[key][item] naming option?


.. py:function:: writeFitData(self, dataPath=None, fName=None, outStem=None, n=None, fType='pickle', ext=None, **kwargs)

   Dump fit data with various backends.





.. py:function:: processedToHDF5(self, dataKey='fits', dataTypes=['dfLong', 'AFxr'], fType='pdHDF', outStem=None, multiFile=False, timeStamp=True, **kwargs)

   Save processed fit data to HDF5.

   Write self.data['fits']['dfLong'] and self.data['fits']['AFxr'] to file.

   Wraps self.writeFitData for processed data types.

   TODO: generalise to arb set of dataTypes and add checks.



.. py:function:: _pickleData(self, fOut)

   Dump self.data to file using Pickle.

   Usual Pickle caveats apply: not recommended for longevity, but handy for checkpoints and backup.



.. py:function:: _writePDData(self, fOut, dataKey='fits', dataType='dfLong')

   Dump item self.data[dataKey][dataType] to file using Pandas.to_hdf().

   This works well for Pandas Dataframes, including complex data.
   Also works for Xarray (via Pandas conversion routine), but may lose attribs.

   Default case will append to file if it exists, so multiple calls will add items (nodes/groups) to hdf store.
   (Note read in via Pandas requires per key reading in this case.)

   See https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_hdf.html

   And https://pandas.pydata.org/docs/user_guide/io.html#hdf5-pytables



.. py:function:: _writeXRData(self, fOut, dataKey='fits', dataType='AFxr', **kwargs)

   Dump item self.data[dataKey][dataType] to file using ep.writeXarray, built on Xarray.to_netcdf()

   This works well for basic Xarray structures.

   For complex data to netCDF need to either set engine='h5netcdf', forceComplex=True, or split to Re + Im groups (currently default for ep.writeXarray).
   For HDF5 complex data is supported.

   Attributes may also need to be sanitised for netCDF writer.

   See :py:func:`epsproc.IO.writeXarray`__ and https://docs.xarray.dev/en/latest/user-guide/io.html

   For docs on ePSproc Xarray IO (June 2022): https://epsproc.readthedocs.io/en/dev/dataStructures/ePSproc_dataStructures_IO_demo_280622.html

   Currently supported methods/engines:

   - netCDF via 'h5netcdf', 'scipy' or 'netcdf4' engines (using Xarray.to_netcdf())
   - 'hdf5' via h5py (via dictionary conversion routines, see :py:func:`epsproc.ioBackends.hdf5IO.writeXarrayToHDF5`__)



.. py:function:: getFilesList(fileIn=None, fileBase=None, fType='pickle', verbose=False)

   Thin wrapper for epsproc.IO.getFiles - get file list from dir (no subdirs) by type


.. py:function:: loadFitData(self, fList=None, dataPath=None, batch=None, **kwargs)

   Load data dumps from a file or set of files (and stack).

   Currently Pickle files only.

   See writeFitData for other options/file types - to be added here too.

   NOTE: currently only supports single dir for dir scan.
   For file lists can pass names only, in which case dataPath will be added, or pass full paths in list.