the difference between plot from saved csv file vs. plot as displayed in web

[ChrysLiang]

Hi warrenarea,

It is really a nice project, given access to raw data from EEG headset.

Besides visualization of raw EEG data from the plot in web is useful, direct manipulation of raw EEG data is also very important for BCI research.

I saved some raw EEG data using the API to a CSV file. However, after some inspection, I found the data saved into the CSV file is very different from the one being plotted in web (in some way?).

Here is the example (they are roughly synchronized):

It seems to me the plot in web is quite correct (I shake gently with the headset), while the plot through the data as saved in CSV is not realistic.

I believe the data itself should be correctly fetched from EEG headset. However, I suspected that, due to some (running environment, formatting, python vs. web?) factor, the data saved in CSV file not the one we want, although it is correct in some way.

Could you please give me some hints on this discrepancy? What could be cause for this issue?

[warrenarea]

Hello,

I think I can clear up a lot of confusion in this matter, the screenshots here definitely help me to see
what is the issue(s).

First, the CyKIT web rendering/plotting of the data, is not quite correct. You will notice that the data
in the EEG always goes downward on the cartesian plane. This is due in part to two reasons... the first,
is that in order to get all the data to be plotted on their respective lines... I have to acquire a 'baseline'
or an 'average' of the data... and then that gets subtracted, which would be equivalent to Emotiv's
"high-pass filter" function. Which is why, if you look at one of Emotiv's programs, like Xavier, the
second you flip on the headset... the data plot starts down at the very bottom and then levels back
out in a bell curve. This is evidence of their program acquiring a baseline/average of the data.

Currently CyKIT has been taking baseline samples almost constantly... which is in part, why the data
is not being rendered properly. The second reason its not being displayed/rendered properly, is that
in addition to the 'high pass filtering' of the data, I have to also add another number, which would
represent the 'full scale' of the microvolt(uv) measurement. Where full scale is the maximum range of
how many microvolts could be measured, per contact.

I've corrected these visual issues in the next version.
Though I have gotten delayed with releasing the next version because I was interested in
adding more features, and verifying each device is working properly.

Your second screenshot... I see you have selected a large volume of data, and I am guessing
that is what is being plotted on your graph..... However, I also notice, in Column B, it is alternated
between 16/32 where 16 is for DATA, and 32 is for the MEMS/GYRO. This data would not work
together, and might be the source of some confusion.

That is one of the issues I will be addressing in the future, is the recording of EEG data, vs Gyro data,
This issue has happened with another person, where they recorded a CSV file, and it did not render
properly in another application, because it was also including the gyro data.

I have not added a flag yet to sift through EEG vs Gyro, so until then, I can only suggest turning off
the MEMS(gyro) feature. or exclude the B['32'] columns in Excel.

In the new version, I will also have it do the baseline / high pass filter, from Python, instead of having
Javascript do it. This might be faster, as well as give more flexibility.

Before my next release though, I at least want to separate the Javascript scripts based on device....
I think this will greatly improve the speed, and clean up the code, making it a little easier to understand.

[warrenarea]

one thing you can try, is after changing the headset mode to receive data only. (mems off)
you could give the flags nocounter+nobattery

then you can try to save in CSV format again... and then load the CSV into one of the emotiv
products to display the rendering of the data. though until i upload the fixed version, it might not
look quite the same.

[warrenarea]

New version (being worked on)

[ChrysLiang]

Thanks warrenarea, your reply help me solve the csv data file puzzle.

After filtering out all rows containing 32 in Col B, I get a clear plot now, with regular shaking pattern I did when I recorded the data.

[warrenarea]

yes, that looks much better..... one thing you might even try,

running one of Emotiv's programs like Xavier.Pure (if you have it)

and first start recording the CSV in Emotiv, and then a few seconds later,
take a screenshot of the Emotiv program.... and stop the recording.

Then you could compare how close the Excel graph looks, compared to the Emotiv image.

I've done similar experiments, running the CSV recorded files through Xavier, and the
rendering seems to match pretty closely.

[warrenarea]

The CSV files have to be recorded without the Gyros, and has to have additional flags, to remove
battery level and counter. (else the program will crash Xavier)

[speaklessba]

Sorry, I see that the compressed package in the website cannot be downloaded, can you please send me the compressed package of cykit2? Thank you very much