pemtk.fit._io

Module Contents

Functions

setTimeStampedFileName(self[, outStem, n, ext, ...])	Set unique filename as f'{outStem}_n{n}_{timeString.strftime("%d%m%y_%H-%M-%S")}.{ext}'
writeFitData(self[, dataPath, fName, outStem, n, ...])	Dump fit data with various backends.
processedToHDF5(self[, dataKey, dataTypes, fType, ...])	Save processed fit data to HDF5.
_pickleData(self, fOut)	Dump self.data to file using Pickle.
_writePDData(self, fOut[, dataKey, dataType])	Dump item self.data[dataKey][dataType] to file using Pandas.to_hdf().
_writeXRData(self, fOut[, dataKey, dataType])	Dump item self.data[dataKey][dataType] to file using ep.writeXarray, built on Xarray.to_netcdf()
getFilesList([fileIn, fileBase, fType, verbose])	Thin wrapper for epsproc.IO.getFiles - get file list from dir (no subdirs) by type
loadFitData(self[, fList, dataPath, batch])	Load data dumps from a file or set of files (and stack).

pemtk.fit._io.setTimeStampedFileName(self, outStem=None, n=None, ext='pickle', timeString=None, timeFormat='%d%m%y_%H-%M-%S')

Set unique filename as f’{outStem}_n{n}_{timeString.strftime(“%d%m%y_%H-%M-%S”)}.{ext}’

Parameters:

• outStem (str, optional, default = None) – Stem for output file. If None, set to ‘PEMtk_data_dump’

• n (int, optional, default = None) – Int index to include in file name. If None, this will be omitted.

• ext (str, optional, default = 'pickle') – File ending.

• timeString (Datatime object, default = None) – Timestamp for the file. If None, current time will be used.

• timeFormat (Datatime format string, optional, default = "%d%m%y_%H-%M-%S") –

• TODO (additional formatting options, data[key][item] naming option?) –

pemtk.fit._io.writeFitData(self, dataPath=None, fName=None, outStem=None, n=None, fType='pickle', ext=None, **kwargs)

Dump fit data with various backends.

pemtk.fit._io.processedToHDF5(self, dataKey='fits', dataTypes=['dfLong', 'AFxr'], fType='pdHDF', outStem=None, multiFile=False, timeStamp=True, **kwargs)

Save processed fit data to HDF5.

Write self.data[‘fits’][‘dfLong’] and self.data[‘fits’][‘AFxr’] to file.

Wraps self.writeFitData for processed data types.

TODO: generalise to arb set of dataTypes and add checks.

pemtk.fit._io._pickleData(self, fOut)

Dump self.data to file using Pickle.

Usual Pickle caveats apply: not recommended for longevity, but handy for checkpoints and backup.

pemtk.fit._io._writePDData(self, fOut, dataKey='fits', dataType='dfLong')

Dump item self.data[dataKey][dataType] to file using Pandas.to_hdf().

This works well for Pandas Dataframes, including complex data. Also works for Xarray (via Pandas conversion routine), but may lose attribs.

Default case will append to file if it exists, so multiple calls will add items (nodes/groups) to hdf store. (Note read in via Pandas requires per key reading in this case.)

See https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_hdf.html

And https://pandas.pydata.org/docs/user_guide/io.html#hdf5-pytables

pemtk.fit._io._writeXRData(self, fOut, dataKey='fits', dataType='AFxr', **kwargs)

Dump item self.data[dataKey][dataType] to file using ep.writeXarray, built on Xarray.to_netcdf()

This works well for basic Xarray structures.

For complex data to netCDF need to either set engine=’h5netcdf’, forceComplex=True, or split to Re + Im groups (currently default for ep.writeXarray). For HDF5 complex data is supported.

Attributes may also need to be sanitised for netCDF writer.

See :py:func:`epsproc.IO.writeXarray`__ and https://docs.xarray.dev/en/latest/user-guide/io.html

For docs on ePSproc Xarray IO (June 2022): https://epsproc.readthedocs.io/en/dev/dataStructures/ePSproc_dataStructures_IO_demo_280622.html

Currently supported methods/engines:

• netCDF via ‘h5netcdf’, ‘scipy’ or ‘netcdf4’ engines (using Xarray.to_netcdf())

• ‘hdf5’ via h5py (via dictionary conversion routines, see :py:func:`epsproc.ioBackends.hdf5IO.writeXarrayToHDF5`__)

pemtk.fit._io.getFilesList(fileIn=None, fileBase=None, fType='pickle', verbose=False)

Thin wrapper for epsproc.IO.getFiles - get file list from dir (no subdirs) by type

pemtk.fit._io.loadFitData(self, fList=None, dataPath=None, batch=None, **kwargs)

Load data dumps from a file or set of files (and stack).

Currently Pickle files only.

See writeFitData for other options/file types - to be added here too.

NOTE: currently only supports single dir for dir scan. For file lists can pass names only, in which case dataPath will be added, or pass full paths in list.