glhmm.preproc

Preprocessing functions - General/Gaussian Linear Hidden Markov Model @author: Diego Vidaurre 2023

glhmm.preproc.apply_ica(X, d, algorithm='parallel')

Applies ICA to the input data X.

Parameters:

Xarray-like of shape (n_samples, n_parcels)

The input data to be transformed.

dint or float

If int, the number of components to keep. If float, the percentage of explained variance to keep (according to a PCA decomposition)

algorithm{“parallel”, “deflation”}, default=”parallel”

Specify which algorithm to use for FastICA.

Returns:

Xarray-like of shape (n_samples, n_components)

The transformed data after applying ICA.

icamodesklearn estimator

The estimated ICA model

glhmm.preproc.apply_pca(X, d, exact=True, whitening=False, center=False)

Applies PCA to the input data X.

Parameters:

Xarray-like of shape (n_samples, n_parcels)

The input data to be transformed.

dint or float

If int, the number of components to keep. If float, the percentage of explained variance to keep. If array-like of shape (n_parcels, n_components), the transformation matrix.

exactbool, default=True

Whether to use full SVD solver for PCA

whiteningbool, default=False

Whether to apply whitening of PCs

centerbool, default=False

Whether to center signal

Returns:

Xarray-like of shape (n_samples, n_components)

The transformed data after applying PCA.

pcamodelsklearn estimator

The estimated PCA model

glhmm.preproc.build_data_autoregressive(data, indices, autoregressive_order=1, connectivity=None, center_data=True, files=None, output_dir=None, file_name=None)

Builds X and Y for the autoregressive model. Supports both in-memory and file-based input. Saves output when processing files.

Parameters:

datandarray, shape (n_samples, n_parcels)

In-memory time series data.

indicesndarray, shape (n_sessions, 2)

Session boundaries in data.

autoregressive_orderint

Number of lags to include.

connectivityndarray, optional

Mask of shape (n_parcels, n_parcels).

center_databool

Whether to mean-center X and Y.

fileslist of str or Path, optional

Input .npz or .mat files to process.

output_dirstr or Path, optional

Directory to save processed files.

file_namestr, optional

Custom suffix to append to each output file name.

Returns:

Xarray-like of shape (n_samples - n_sessions*autoregressive_order, n_parcels*autoregressive_order)

The timeseries of set of variables 1 (i.e., the regressors).

Yarray-like of shape (n_samples - n_sessions*autoregressive_order, n_parcels)

The timeseries of set of variables 2 (i.e., variables to predict, targets).

indices_newarray-like of shape (n_sessions, 2)

The new array of start and end indices for each trial/session.

connectivity_newarray-like of shape (n_parcels*autoregressive_order, n_parcels)

The new connectivity matrix indicating which regressors should be used for each variable.

glhmm.preproc.build_data_partial_connectivity(X, Y, connectivity=None, center_data=True)

Builds X and Y for the partial connectivity model, essentially regressing out things when indicated in connectivity, and getting rid of regressors / regressed variables that are not used; it return connectivity with the right dimensions as well.

Parameters:

Xnp.ndarray of shape (n_samples, n_parcels)

The timeseries of set of variables 1 (i.e., the regressors).

Ynp.ndarray of shape (n_samples, n_parcels)

The timeseries of set of variables 2 (i.e., variables to predict, targets).

connectivitynp.ndarray of shape (n_parcels, n_parcels), optional, default=None

A binary matrix indicating which regressors affect which targets (i.e., variables to predict).

center_databool, default=True

Center data to zero mean.

Returns:

X_newnp.ndarray of shape (n_samples, n_active_parcels)

The timeseries of set of variables 1 (i.e., the regressors) after removing unused predictors and regressing out the effects indicated in connectivity.

Y_newnp.ndarray of shape (n_samples, n_active_parcels)

The timeseries of set of variables 2 (i.e., variables to predict, targets) after removing unused targets and regressing out the effects indicated in connectivity.

connectivity_newnp.ndarray of shape (n_active_parcels, n_active_parcels), optional, default=None

A binary matrix indicating which regressors affect which targets The matrix has the same structure as connectivity after removing unused predictors and targets.

glhmm.preproc.build_data_tde(data=None, indices=None, lags=None, pca=None, standardise_pc=True, files=None, output_dir=None, file_name=None)

Builds delay-embedded data for TDE-HMM. Supports in-memory or file-based input.

Parameters:

datandarray or None

Raw data (n_samples, n_parcels) to embed in memory.

indicesndarray or None

Start and end indices for each session (n_sessions, 2).

lagslist or array-like

Lags to apply for temporal embedding.

pcaint, float, array or None

PCA options.

standardise_pcbool

Whether to standardise PCA components.

fileslist of str or Path, optional

If set, reads files instead of using data/indices.

output_dirstr or Path, optional

Where to save output files if using file input.

file_namestr or None, optional

Custom string to append to each output file name before extension.

Returns:

If using files: list of output file paths, and log dictionary if PCA is applied. If using in-memory data: X_emb, indices_emb (+ pcamodel if PCA).

glhmm.preproc.compute_unique_suffixes(paths)

glhmm.preproc.dampen_peaks(X, strength=5)

Applies dampening of extreme peaks to the input data X, at the group level. If the absolute value of X goes beyond 2 standard deviation of X, it gets substituted by the logarithm of the absolute value of X.

Parameters:

Xarray-like of shape (n_samples, n_parcels)

The input data to be transformed.

strengthpositive int

The strength of dampening. This value refers to the base of the logarithm to use. The bigger the base, the stronger the dampening.

Returns:

X_transformedarray-like of shape (n_samples, n_parcels)

The transformed data after applying extreme peak dampening.

glhmm.preproc.highdim_pca(C, n_components=None)

Perform PCA on a high-dimensional correlation or covariance matrix.

Parameters:

Cndarray of shape (p, p)

The input correlation or covariance matrix.

n_componentsint or float or None

Number of components or proportion of explained variance to retain.

Returns:

eigvecsndarray of shape (p, n_components)

The principal component directions.

eigvalsndarray of shape (n_components,)

The corresponding eigenvalues indicating variance explained.

glhmm.preproc.load_X(file_path)

Load a data array from a file.

Parameters:

INPUT_FILE_PATHstr or Path

Path to the input file (.npy, .npz, .mat, or .txt).

Returns:

Xndarray of shape (n_samples, n_features)

The loaded data array, reshaped to 2D if needed.

glhmm.preproc.load_files(files, I=None, do_only_indices=False)

glhmm.preproc.preprocess_data(data=None, indices=None, fs=1, dampen_extreme_peaks=None, standardise=True, filter=None, detrend=False, onpower=False, onphase=False, pca=None, exact_pca=True, whitening=False, center=False, ica=None, ica_algorithm='parallel', post_standardise=None, downsample=None, files=None, combine_outputs=True, combined_name=None, output_dir=None, file_name=None, file_type='npy', lags=None, autoregressive_order=None)

Preprocess the input data or files, with support for stochastic training and optional TDE embedding.

Parameters:

dataarray-like, optional

Raw input data of shape (n_samples, n_parcels), used for in-memory processing.

indicesarray-like of shape (n_sessions, 2), optional

Start and end indices of each session (only required for in-memory data).

fsint or float, default=1

Sampling frequency of the data.

dampen_extreme_peaksint, bool, or None, default=None

Dampens extreme peaks in the data. If True, uses default strength of 5. If int, specifies strength.

standardisebool, default=True

Whether to standardise (zero-mean, unit-variance) each session.

filtertuple of two floats or None, default=None

Bandpass (low, high), lowpass (0, high), or highpass (low, None) filter.

detrendbool, default=False

Whether to linearly detrend the data.

onpowerbool, default=False

Whether to extract signal power using the Hilbert transform.

onphasebool, default=False

Whether to extract phase using the Hilbert transform. If both onpower and onphase are True, power and phase are concatenated.

pcaint, float, array-like, or None, default=None

PCA dimensionality reduction. If int, number of components. If float, proportion of variance to retain. If array, treated as precomputed PCA matrix.

exact_pcabool, default=True

Whether to use full SVD in PCA (only relevant for in-memory mode).

whiteningbool, default=False

Whether to apply whitening of PCs

centerbool, default=False

Whether to center signal before applying PCA

icaint or float or None, default=None

ICA dimensionality reduction. If int, number of components. If float, proportion of variance to retain.

ica_algorithmstr, default=’parallel’

ICA algorithm to use (e.g., ‘parallel’, ‘deflation’).

post_standardisebool or None, default=None

Whether to standardise data after PCA or ICA. Defaults to True if ICA is used.

downsampleint or float or None, default=None

New sampling frequency. If None, no downsampling.

fileslist of str or Path, optional

If set, enables file-based preprocessing with one file at a time for stochastic training.

combine_outputsbool, default=True

When using file inputs whether to write output as individual files or combine the output

combined_namestr, default=”combined”

Stem for output file name (only when using files and combine_outputs=True)

output_dirstr or Path, optional

Directory to save processed files (only used in file mode).

file_namestr or None, optional

Optional suffix to append to output filenames.

file_typestr, default=”npy”

File format type for loading files.

lagslist, optional

If specified, applies temporal delay embedding (TDE) using these lags. This prepares the data for use with a Time-Delay Embedded HMM (HMM-TDE). This should be a list of integers indicating how many time steps before and after to include. For example, use:

lags = np.arange(-7, 8)

to include 15 lagged versions of the signal: from 7 time steps before to 7 time steps after.

autoregressive_orderint, default=None

Number of lags to include.

Returns:

For in-memory mode:

datanp.ndarray

The preprocessed (and optionally embedded/reduced) data.

indices_newnp.ndarray

Updated indices after preprocessing and embedding.

logdict

Dictionary containing the preprocessing parameters and models.

For file-based mode:

output_file_pathslist of str

List of paths to saved preprocessed files.

logdict

Dictionary with accumulated preprocessing parameters and PCA statistics.

glhmm.preproc.resolve_files(files, file_type='npz')