Accuracy Loss at Lower Sampling Frequencies

[StantonGodshall]

Hello,

I apologize if this question should be asked elsewhere. I have found that if I downsample my 100Hz data to 15Hz, there is a substantial reduction in the total number of steps detected (36% reduction). I know that the original paper validated the algorithm on both 100Hz and 15Hz datasets, so I just wanted to confirm that this loss is expected and not attributable to something else. I understand there are inherent challenges to sampling at lower rates. We have a cohort of patients with low sampling frequencies (12.5Hz) and with high frequency (100Hz), so I was hoping to compare them. I'm happy to share more details/data files if necessary.

Thank you,
Stanton

[chanshing]

That's quite a reduction. Can you share how you are downsampling the data?

[StantonGodshall]

Yes. some context I can add as well:

If I reduce the data to 25Hz the results don't change appreciably. I do this by just taking every 4th point of the original data.
When downsampling to 15Hz I use scipy.resample.

[chanshing]

OK that's useful info. I think scipy.resample applies some filtering which might be affecting the results. Can you try again using https://actipy.readthedocs.io/en/latest/api.html#actipy.processing.resample ? This another pkg we maintain - it applies a less aggressive downsampling

[StantonGodshall]

I got the same result. downsampling to 15 Hz (with actipy.processing.resample) reduces the total number of steps substantially. for QC, I used this method and downsampled to 25Hz as well and found that accuracy is maintained relative to the 100Hz data.

[chanshing]

To further help me debug this, can you try the random forest model (--t rf) and see difference between 15 Hz and 100 Hz?

[StantonGodshall]

Yes. The original model estimated ~145000 steps at 100Hz. the random forest model estimated 121000 steps at 100Hz and 138000 at 15Hz.

[chanshing]

How many steps at 15Hz using the original model?

[StantonGodshall]

89k

[chanshing]

OK to summarize:

SSL model (default):

• At 100Hz: ~145k
• At 15Hz: ~89k

RF model (--t rf):

• At 100Hz: ~121k
• At 15Hz: ~138k

Is that right?

I can sort of imagine why RF would result in more steps when undersampled (aliasing causing spiky patterns perhaps) but no clue for SSL. One guess is SSL rejecting the segments as walking altogether due to the same aliasing issues (RF extracts features that are a bit more robust to these).

If so, this is a harder problem to solve that would require either a) fine-tuning the SSL-based walking classifier (easier), or b) retraining the underlying SSL foundation model (hard)

@aidanacquah @angerhang just tagging you for visibility and in case you have any thoughts

[StantonGodshall]

Yes exactly correct. Thanks for taking the time to look into it

[angerhang]

OK to summarize:

SSL model (default):

• At 100Hz: ~145k
• At 15Hz: ~89k

RF model (--t rf):

• At 100Hz: ~121k
• At 15Hz: ~138k

Is that right?

I can sort of imagine why RF would result in more steps when undersampled (aliasing causing spiky patterns perhaps) but no clue for SSL. One guess is SSL rejecting the segments as walking due to the same aliasing issues (RF extracts features that are a bit more robust to these).

If so, this is a harder problem to solve that would require either a) fine-tuning the SSL-based walking classifier (easier), or b) retraining the underlying SSL foundation model (hard)

@aidanacquah @angerhang just tagging you for visibility and in case you have any thoughts

Thanks both, I wasn't aware that the fine-tuning stage was done on 15hz as the pre-trained model only saw 30hz of data.

I'd imagine for lower frequency input, the data is up sampled to 30hz?

To me, we shouldn't really support inference for SSL model using different sample rates. Perhaps it is meaningful to include an assert statement to prevent future use of such kind.

RF models should be preferred using non-30hz data. We don't have the capacity to pre-train the SSL model using different sampling rates even though it is technically possible. But perhaps fine-tuning could be done.

[chanshing]

Thanks @angerhang for your input
To clarify, the walking detection layer was indeed trained on 30Hz data - not 15Hz. I meant to say we could probably train the walking detection layer to handle various sampling rates (perhaps by including decimation + upsampling as augmentation). Just a thought!

[aidanacquah]

Implement decimation augmentation as seen in: OxWearables/actinet#33