06/12/21 - TO FIX: push/debug ePSproc updates, currently throwing loading errors on Jake

Analysis routines

  • 06/12/21 v2 with updated param analysis and tabulation routines.

  • 14/11/21 v1 (still in development)

In this notebook some of the packaged analysis routines are demonstrated. For an earlier exploration of the basic/underlying analysis routines, see the previous fit fidelity & analysis notebook.

Setup

Load fit & analysis class and import data (currently a bit messy).

[1]:
# Import & set paths
import pemtk
from pemtk.fit.fitClass import pemtkFit

from pathlib import Path

# Path for demo script
demoPath = Path(pemtk.__file__).parent.parent/Path('demos','fitting')

# Some additional default plot settings
# TODO: this is already run at class init, but out of notebook scope? Should fix.
from epsproc.plot import hvPlotters
hvPlotters.setPlotters()
*** ePSproc installation not found, setting for local copy.
C:\Users\femtolab\.conda\envs\ePSdev\lib\importlib\_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
  return f(*args, **kwds)
C:\Users\femtolab\.conda\envs\ePSdev\lib\site-packages\xyzpy\plot\xyz_cmaps.py:6: MatplotlibDeprecationWarning:
The revcmap function was deprecated in Matplotlib 3.2 and will be removed two minor releases later. Use Colormap.reversed() instead.
  return LinearSegmentedColormap(name, cm.revcmap(cmap._segmentdata))
[2]:
# Here we'll just load some test data
# TODO: wrap this to class!

# Load sample dataset
# Full path to the file may be required here, in repo/demos/fitting
import pickle
from pathlib import Path

# dataPath = Path(pemtk.__path__[0]).parent/Path('demos','fitting')  # Test data in repo
dataPath = demoPath

# Basic test data - simulated results with no noise, 100 fits
# dataFile = 'dataDump_100fitTests_10t_randPhase_130621.pickle'

# Noisey test data - simulated results with noise, 1000 fits
dataFile = 'dataDump_1000fitTests_multiFit_noise_051021.pickle'


# Set for empty class, or with full demo setup (includes ref. parameter set)
demo = False

if demo:
    # Version with full demo setup
    %run {demoPath/"setup_fit_demo.py"}


else:
    data = pemtkFit()  # Data loads OK for blank class, but may be missing some necessary vars.

data.verbose['sub'] = 1
with open( dataPath/dataFile, 'rb') as handle:
    data.data = pickle.load(handle)

data.fitInd = list(data.data.keys())[-1]  # Set final key from data - probably not a robust method however!
                                          # This is currently used for indexing

Data overview

For more on the data structures used here, see the setup & batch runs notebook.

[3]:
# Check number of datasets loaded
data.fitInd
[3]:
999
[4]:
# Run basic stats
# TODO: add more outputs here & tidy up output formatting.
data.analyseFits()
Pandas reference table not set, missing self.params data.
Setting wide-form data self.[fits][dfWide] from self.[fits][dfLong] (as pivot table).
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
Index(es) = ['Fit', 'Type'], cols = value
{ 'Fits': 999,
  'Minima': {'chisqr': 0.04215746268613921, 'redchi': 0.00022911664503336526},
  'Success': 991}
[5]:
# Basic histogram of fit sets
# Note this defaults to Holoviews/Bokeh for plotting, which produces an interactive plot.
# Set backend = 'mpl' if Holoviews is not available.
data.fitHist()

The overview histogram is fairly coarse, since it shows all results. It is clear, however, that there are many results at the low end of the range (by default this plots reduced \(\chi^2\) values). This can be explored in further detail via the interactive plot (if using Holoviews), or via further plotting with specified thresholds or ranges (see below).

Dataset notes:

  • For “perfect” data (dataDump_1000fitTests_multiFit_130921.pickle) can get a large spread here, with best results at \(10^{-11}\).

  • For noisey data (dataDump_1000fitTests_multiFit_noise_011021.pickle) have a smaller spread, best results low \(10^{-4}\).

[6]:
# With threshold set & fine binning.
# See fitHist() docs for more options.
data.fitHist(thres = 2.4e-4, bins = 100)

# data.fitHist(bins=20, binRange = [2.3e-4, 2.4e-4])   # Example with a range set
Mask selected 561 results (from 999).

Data exploration

The general aim in this procedure is to ascertain whether there was a good spread of parameters explored, and a single (or few sets) of best-fit results. There are a few procedures and helper methods for this…

View results

Single results sets can be viewed in the main data structure, indexed by #.

[7]:
# Check keys
fitNumber = 2
data.data[fitNumber].keys()
[7]:
dict_keys(['AFBLM', 'residual', 'results'])

Here results is an lmFit object, which includes final fit results and information, and AFBLM contains the model output. (TODO: helper functions for this.)

An example is shown below. Of particular note here is which parameters have vary=True - these are included in the fitting - and if there is a column expression, which indicates any parameters defined to have specific relationships (see the basic demo notebook for more). Any correlations found during fitting are also shown, which can also indicate parameters which are related (even if this is not predefined or known a priori).

For the current dataset (‘dataDump_1000fitTests_multiFit_noise_051021.pickle’), note that p_PU_SG_PU_1_n1_1_1 has vary=False, hence the results are already defined as relative phases (and, in this example, the fixed value is kept at the original model value).

[8]:
# Show some results
data.data[fitNumber]['results']
[8]:

Fit Statistics

fitting methodleastsq
# function evals2812
# data points195
# variables11
chi-square 0.04215747
reduced chi-square 2.2912e-04
Akaike info crit.-1623.67186
Bayesian info crit.-1587.66886

Variables

name value standard error relative error initial value min max vary
m_PU_SG_PU_1_n1_1_1 1.61758339 227151.843 (14042666.65%) 0.4963375310160162 1.0000e-04 5.00000000 True
m_PU_SG_PU_1_1_n1_1 1.60986876 227266.217 (14117064.81%) 0.349972501184159 1.0000e-04 5.00000000 True
m_PU_SG_PU_3_n1_1_1 1.14366717 214620.511 (18765993.77%) 0.43544003283291643 1.0000e-04 5.00000000 True
m_PU_SG_PU_3_1_n1_1 1.14104307 213989.161 (18753819.79%) 0.16101762796183416 1.0000e-04 5.00000000 True
m_SU_SG_SU_1_0_0_1 2.65253557 0.06899386 (2.60%) 0.8234700291388766 1.0000e-04 5.00000000 True
m_SU_SG_SU_3_0_0_1 1.14378960 0.14902324 (13.03%) 0.5411867072477154 1.0000e-04 5.00000000 True
p_PU_SG_PU_1_n1_1_1 -0.86104140 0.00000000 (0.00%) -0.8610414024232179 -3.14159265 3.14159265 False
p_PU_SG_PU_1_1_n1_1 -0.85729904 167895.839 (19584279.35%) 0.021992539259326538 -3.14159265 3.14159265 True
p_PU_SG_PU_3_n1_1_1 1.20005808 224407.741 (18699740.07%) 0.023782497271738423 -3.14159265 3.14159265 True
p_PU_SG_PU_3_1_n1_1 1.19902071 296532.682 (24731239.48%) 0.5554954153030479 -3.14159265 3.14159265 True
p_SU_SG_SU_1_0_0_1 2.43703747 83742.0262 (3436222.35%) 0.5266569153447359 -3.14159265 3.14159265 True
p_SU_SG_SU_3_0_0_1 -0.70401900 83725.8242 (11892551.81%) 0.004178913362662073 -3.14159265 3.14159265 True

Correlations (unreported correlations are < 0.100)

m_PU_SG_PU_1_n1_1_1m_PU_SG_PU_1_1_n1_1-1.0000
m_PU_SG_PU_3_n1_1_1m_PU_SG_PU_3_1_n1_1-1.0000
p_SU_SG_SU_1_0_0_1p_SU_SG_SU_3_0_0_11.0000
p_PU_SG_PU_1_1_n1_1p_SU_SG_SU_1_0_0_10.9999
p_PU_SG_PU_1_1_n1_1p_SU_SG_SU_3_0_0_10.9999
m_PU_SG_PU_3_n1_1_1p_PU_SG_PU_3_n1_1_10.9923
m_PU_SG_PU_3_1_n1_1p_PU_SG_PU_3_n1_1_1-0.9921
m_PU_SG_PU_1_n1_1_1m_PU_SG_PU_3_1_n1_10.9852
m_PU_SG_PU_1_n1_1_1m_PU_SG_PU_3_n1_1_1-0.9849
m_PU_SG_PU_1_1_n1_1m_PU_SG_PU_3_1_n1_1-0.9848
m_PU_SG_PU_1_1_n1_1m_PU_SG_PU_3_n1_1_10.9845
m_PU_SG_PU_1_n1_1_1p_PU_SG_PU_3_n1_1_1-0.9585
m_PU_SG_PU_1_1_n1_1p_PU_SG_PU_3_n1_1_10.9579
m_SU_SG_SU_1_0_0_1m_SU_SG_SU_3_0_0_1-0.9312
p_PU_SG_PU_3_n1_1_1p_PU_SG_PU_3_1_n1_1-0.8223
m_PU_SG_PU_3_n1_1_1p_PU_SG_PU_3_1_n1_1-0.7514
m_PU_SG_PU_3_1_n1_1p_PU_SG_PU_3_1_n1_10.7503
p_PU_SG_PU_1_1_n1_1p_PU_SG_PU_3_1_n1_10.6663
p_PU_SG_PU_3_1_n1_1p_SU_SG_SU_1_0_0_10.6480
p_PU_SG_PU_3_1_n1_1p_SU_SG_SU_3_0_0_10.6476
m_PU_SG_PU_1_n1_1_1p_PU_SG_PU_3_1_n1_10.6332
m_PU_SG_PU_1_1_n1_1p_PU_SG_PU_3_1_n1_1-0.6316
m_PU_SG_PU_3_n1_1_1m_SU_SG_SU_1_0_0_1-0.5284
m_PU_SG_PU_3_1_n1_1m_SU_SG_SU_1_0_0_10.5284
m_SU_SG_SU_1_0_0_1p_PU_SG_PU_3_n1_1_1-0.5263
m_PU_SG_PU_1_n1_1_1m_SU_SG_SU_1_0_0_10.5166
m_PU_SG_PU_1_1_n1_1m_SU_SG_SU_1_0_0_1-0.5164
m_PU_SG_PU_3_n1_1_1m_SU_SG_SU_3_0_0_10.4968
m_PU_SG_PU_3_1_n1_1m_SU_SG_SU_3_0_0_1-0.4967
m_SU_SG_SU_3_0_0_1p_PU_SG_PU_3_n1_1_10.4949
m_PU_SG_PU_1_n1_1_1m_SU_SG_SU_3_0_0_1-0.4853
m_PU_SG_PU_1_1_n1_1m_SU_SG_SU_3_0_0_10.4851
m_SU_SG_SU_1_0_0_1p_PU_SG_PU_3_1_n1_10.4101
m_SU_SG_SU_3_0_0_1p_PU_SG_PU_3_1_n1_1-0.3865
m_PU_SG_PU_1_1_n1_1p_SU_SG_SU_3_0_0_10.1724
m_PU_SG_PU_1_1_n1_1p_SU_SG_SU_1_0_0_10.1719
m_PU_SG_PU_1_n1_1_1p_SU_SG_SU_3_0_0_1-0.1702
m_PU_SG_PU_1_n1_1_1p_SU_SG_SU_1_0_0_1-0.1697
m_PU_SG_PU_1_1_n1_1p_PU_SG_PU_1_1_n1_10.1670
m_PU_SG_PU_1_n1_1_1p_PU_SG_PU_1_1_n1_1-0.1648
p_PU_SG_PU_1_1_n1_1p_PU_SG_PU_3_n1_1_1-0.1330
p_PU_SG_PU_3_n1_1_1p_SU_SG_SU_1_0_0_1-0.1088
p_PU_SG_PU_3_n1_1_1p_SU_SG_SU_3_0_0_1-0.1083

Classify fits

To review and classify the “sets” of fit results which are apparent here, use the classifyFits() method. This will rebin/categorise the data and label each bin alphabetically.

[9]:
# The default case simply rebins all data into 10 categories.
data.classifyFits()
success chisqr redchi Min Max
count unique top freq count unique top freq count unique top freq
redchiGroup
A 835 2 True 833 835.0 835.0 0.044630 1.0 835.0 835.0 0.000229 1.0 0.000218 0.000321
B 80 2 True 79 80.0 80.0 0.069253 1.0 80.0 80.0 0.000377 1.0 0.000321 0.000424
C 14 2 True 13 14.0 14.0 0.088500 1.0 14.0 14.0 0.000440 1.0 0.000424 0.000527
D 11 1 True 11 11.0 11.0 0.103048 1.0 11.0 11.0 0.000605 1.0 0.000527 0.000630
E 6 1 True 6 6.0 6.0 0.132356 1.0 6.0 6.0 0.000719 1.0 0.000630 0.000733
F 18 1 True 18 18.0 18.0 0.140435 1.0 18.0 18.0 0.000775 1.0 0.000733 0.000836
G 4 1 True 4 4.0 4.0 0.154999 1.0 4.0 4.0 0.000842 1.0 0.000836 0.000939
H 7 1 True 7 7.0 7.0 0.173290 1.0 7.0 7.0 0.000942 1.0 0.000939 0.001042
I 2 1 True 2 2.0 2.0 0.193286 1.0 2.0 2.0 0.001053 1.0 0.001042 0.001146
Setting wide-form data self.[fits][dfWide] from self.[fits][dfLong] (as pivot table).
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
Index(es) = ['Fit', 'Type', 'redchiGroup'], cols = value
Set redchiGroup for data frame dfLong.
Set redchiGroup for data frame AFpdLong.
Couldn't set redchiGroup for data frame mask. Error <class 'KeyError'>: ('dType',).
../_images/fitting_PEMtk_fitting_multiproc_class_analysis_141121-tidy_16_2.png
[10]:
# Again, more control can be obtained by specifiying the desired binning.
data.classifyFits(bins = [2.26e-4, 2.4e-4,20])
success chisqr redchi Min Max
count unique top freq count unique top freq count unique top freq
redchiGroup
A 0 0 NaN NaN 0 0 NaN NaN 0 0 NaN NaN 0.000226 0.000227
B 0 0 NaN NaN 0 0 NaN NaN 0 0 NaN NaN 0.000227 0.000227
C 0 0 NaN NaN 0 0 NaN NaN 0 0 NaN NaN 0.000227 0.000228
D 0 0 NaN NaN 0 0 NaN NaN 0 0 NaN NaN 0.000228 0.000229
E 349 1 True 349 349 349 0.0421575 1 349 349 0.000229117 1 0.000229 0.000230
F 0 0 NaN NaN 0 0 NaN NaN 0 0 NaN NaN 0.000230 0.000230
G 27 1 True 27 27 27 0.0425252 1 27 27 0.000231115 1 0.000230 0.000231
H 43 1 True 43 43 43 0.0426516 1 43 43 0.000231163 1 0.000231 0.000232
I 24 1 True 24 24 24 0.0426877 1 24 24 0.000231998 1 0.000232 0.000233
J 0 0 NaN NaN 0 0 NaN NaN 0 0 NaN NaN 0.000233 0.000233
K 11 1 True 11 11 11 0.0430736 1 11 11 0.000234096 1 0.000233 0.000234
L 27 1 True 27 27 27 0.0432038 1 27 27 0.000234803 1 0.000234 0.000235
M 3 2 True 2 3 3 0.0433464 1 3 3 0.000235579 1 0.000235 0.000236
N 54 1 True 54 54 54 0.0433468 1 54 54 0.000235581 1 0.000236 0.000236
O 5 1 True 5 5 5 0.0435333 1 5 5 0.00023659 1 0.000236 0.000237
P 0 0 NaN NaN 0 0 NaN NaN 0 0 NaN NaN 0.000237 0.000238
Q 15 2 True 14 15 15 0.0438441 1 15 15 0.000237988 1 0.000238 0.000239
R 3 1 True 3 3 3 0.0439158 1 3 3 0.000238673 1 0.000239 0.000239
S 0 0 NaN NaN 0 0 NaN NaN 0 0 NaN NaN 0.000239 0.000240
Setting wide-form data self.[fits][dfWide] from self.[fits][dfLong] (as pivot table).
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
Index(es) = ['Fit', 'Type', 'redchiGroup'], cols = value
Set redchiGroup for data frame dfLong.
Set redchiGroup for data frame AFpdLong.
Couldn't set redchiGroup for data frame mask. Error <class 'KeyError'>: ('dType',).
../_images/fitting_PEMtk_fitting_multiproc_class_analysis_141121-tidy_17_2.png

In this case, there are 349 fit results within the lowest (occupied) category (set E), which we can take to be the best fits possible in this case.

To visualise the associated parameter sets, we can use the corrPlot() and paramPlot() methods. These default to the currently set classified data (this is found in the wide-form data output by classifyFits(), which defaults to self.[fits][dfWide]), with colour-coding by group.

corrPlot()

corrPlot() uses Seaborn’s pairplot routine to generate a full pair-wise correlation plot.

TODO: add HV gridmatrix + linked brushing: http://holoviews.org/user_guide/Linked_Brushing.html

[11]:
# Default plot will give output by grouping
data.corrPlot()
Mask not set for dataType = None.
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
<seaborn.axisgrid.PairGrid at 0x20519f5e048>
../_images/fitting_PEMtk_fitting_multiproc_class_analysis_141121-tidy_20_2.png
[12]:
# For more control, standard catplot parameters can be passed.
# See https://seaborn.pydata.org/generated/seaborn.pairplot.html

# Plot with colour by Type and histograms on the diagonal
data.corrPlot(hue='Type', diag_kind='hist')

# Additional per-fit data can also be set for the hue mapping,
# e.g. plot with colour by redchi (to 6 dp) and histograms on the diagonal
# data.corrPlot(hue='redchi', hRound = 6, diag_kind='hist')
Mask not set for dataType = None.
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
<seaborn.axisgrid.PairGrid at 0x2051b21eeb8>
../_images/fitting_PEMtk_fitting_multiproc_class_analysis_141121-tidy_21_2.png
[13]:
# Tabulated output from the last plot can be found in self.data['plots']['<plotType>Data'], and the plot object in self.data['plots']['<plotType>Plot']
data.data['plots']['corrData']
[13]:
Param Type redchiGroup PU_SG_PU_1_1_n1_1 PU_SG_PU_1_n1_1_1 PU_SG_PU_3_1_n1_1 PU_SG_PU_3_n1_1_1 SU_SG_SU_1_0_0_1 SU_SG_SU_3_0_0_1
0 m Q 1.992972 1.921566 0.466055 0.000100 2.461439 1.556832
1 p Q -0.861561 -0.861041 3.141575 1.313656 -3.032645 -0.647618
2 m E 1.609869 1.617583 1.141043 1.143667 2.652536 1.143790
3 p E -0.857299 -0.861041 1.199021 1.200058 2.437037 -0.704019
4 m K 1.932782 1.931857 0.355341 0.368418 2.533506 1.418030
... ... ... ... ... ... ... ... ...
1117 p L 0.145056 -0.861041 2.001652 1.692756 -3.141593 0.230230
1118 m N 1.952031 1.992561 0.000100 0.805098 2.387993 1.651091
1119 p N -0.739526 -0.861041 -1.434696 0.937920 1.420065 -0.905355
1120 m E 1.613212 1.614101 1.142782 1.142106 2.652523 1.143815
1121 p E -0.862937 -0.861041 1.195923 1.197572 2.434125 -0.708249

1122 rows × 8 columns

paramPlot()

For examining specific datatypes in more detail paramPlot() uses Seaborn’s catplot routine to generate scatter plots for a specified data type.

[14]:
# Plot all data, magnitudes
data.paramPlot(dataType='m')
Mask not set for dataType = None.
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
<seaborn.axisgrid.FacetGrid at 0x2051e7fa240>
../_images/fitting_PEMtk_fitting_multiproc_class_analysis_141121-tidy_24_2.png
[15]:
# Plot all data, phases
data.paramPlot(dataType='p')
Mask not set for dataType = None.
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
<seaborn.axisgrid.FacetGrid at 0x2051e9c09e8>
../_images/fitting_PEMtk_fitting_multiproc_class_analysis_141121-tidy_25_2.png

Here we can see that there are various sets of parameters which do, indeed, cluster by group (hence \(\chi^2\)) for the most part, although there is a full spread in some of the phase parameters. We can look more closely at this by selecting just the best group(s) and colour-coding by \(\chi^2\)

[16]:
# Subselect on redchiGroup and colour by redchi value (note this automatically rounds to 7 dp, set hRound = N to control this)
data.paramPlot(dataType='m', sel = 'E', hue = 'redchi')
Mask not set for dataType = None.
<seaborn.axisgrid.FacetGrid at 0x2051eb04c50>
../_images/fitting_PEMtk_fitting_multiproc_class_analysis_141121-tidy_27_2.png
[17]:
# As above, but for phase data
data.paramPlot(dataType='p', sel = 'E', hue = 'redchi')
Mask not set for dataType = None.
<seaborn.axisgrid.FacetGrid at 0x2051fd0bdd8>
../_images/fitting_PEMtk_fitting_multiproc_class_analysis_141121-tidy_28_2.png

In this case, there are clear sub-groupings by \(\chi^2\). For the phases, these can potentially be cleaned up a bit by setting a reference phase & wrapping the values.

[18]:
# Tabulated output from the last plot can be found in self.data['plots']['<plotType>Data'], and the plot object in self.data['plots']['<plotType>Plot']
data.data['plots']['paramData']
[18]:
Fit Param value redchi
0 2 PU_SG_PU_1_1_n1_1 -0.857299 0.000229
1 8 PU_SG_PU_1_1_n1_1 -1.057287 0.000229
2 9 PU_SG_PU_1_1_n1_1 -1.303957 0.000229
3 11 PU_SG_PU_1_1_n1_1 -0.862917 0.000229
4 13 PU_SG_PU_1_1_n1_1 -0.863154 0.000229
... ... ... ... ...
2089 984 SU_SG_SU_3_0_0_1 -0.702315 0.000229
2090 987 SU_SG_SU_3_0_0_1 -0.704560 0.000229
2091 988 SU_SG_SU_3_0_0_1 -1.015555 0.000229
2092 992 SU_SG_SU_3_0_0_1 -0.710594 0.000229
2093 997 SU_SG_SU_3_0_0_1 -0.708249 0.000229

2094 rows × 4 columns

[19]:
# Basic Holoviews support is also working, with a few additional plot options
# data.paramPlot(dataType='p', backend='hv')
data.paramPlot(dataType='p', backend='hv', hvType='violin')   # Include violin KDE
Mask not set for dataType = None.
*** Warning: found MultiIndex for DataFrame data.index - checkDims doesn't yet support Pandas MultiIndex.
[20]:
# Single group with 'redchi' cmap
data.paramPlot(dataType='p', sel = 'E', hue = 'redchi', backend='hv', hvType='violin')
# data.paramPlot(dataType='p', sel = 'E', backend='hv')  # NOTE: this currently doesn't work without Hue set to a numerical category.
Mask not set for dataType = None.