GPS raw data error

[MMel099]

I have identified a new error with GPS data that can cause downstream errors with the Jasmine analysis. When generating the raw data GPS csv files, some rows inexplicably get duplicated in a way that messes up the format (screenshot below). Starting from row 243, rows get duplicated in a way that the duplicates have more columns than the original rows. This then causes a "ParserError: Error tokenizing data. C error: Expected 6 fields in line XXX, saw 7" when running Jasmine.

One way to fix this would be to iterate over all the GPS files, searching and pruning rows with duplicate timestamps. I am starting to look into Mano for the source of the problem but the code is quite advanced.

[MMel099]

@GeorgeEfstathiadis have you seen anything like this before?

[GeorgeEfstathiadis]

No I have not seen such an issue with raw GPS data before, but this feels like a bug with data collection. Maybe @biblicabeebli would know what this is about?

[biblicabeebli]

It's a data processing bug.
I can't find it, I made an issue and assigned everyone on it about these extra columns getting injected but that I thought I had fixed it.
I don't know what's up that's causing it to not even have commas there, that's really strange but also so is Excel so who knows.

Gimme participant ids and their server, I will clear and rerun all of the affected user's data to clear it out. Hard to say exactly how long that takes, but definitely by the next day.

[MMel099]

I did a big Forest task on all of the data in the Yale study. A few participants had this error in both Jasmine and Oak so accelerometer is likely also affected. txo6lahs is my personal id in the Yale study and the first injected row happened in the file from 03/23/24 at 22:00:00 (started collecting data in January on this id).
study
IDs: l2e3gzih, gq4s82qs, mhwnmw12, txo6lahs, ra5aq9d5, e15hkfui

[biblicabeebli]

Just kicked off the reprocessing for those users.

buuuuuut I screwed it up and unintentionally removed identifiers..... I'll go fix that...

[biblicabeebli]

ok those are added too.

[MMel099]

Looks like it worked! I created a forest task for Oak and Jasmine on txo6lahs which is my id and it worked. I'll try the others locally. Quick question - do you know why some Forest tasks on the dashboard have a "download output data" button and others do not?

Also Yale team just pointed out something with Beiwe id ra5aq9d5, one of the ones affected by the issue above. Yesterday they said that this participant did not have any data after 02/26 even though it had been there before. Today, I see more data in the dashboard but still no survey answers/timings. Is this related to the cleaning for the injection issue? Anything else weird with this id? This is one of the few androids in their study.

Thanks

[biblicabeebli]

I will have to look into that participant, the only concerns I can think of would be with iOS participants.

Download data link - when I started typing I thought I knew the answer, but I don't. I will have to check this as well.

[MMel099]

I think the data for id ra5aq9d5 came in a little slowly but is all there now. We can probably close this issue. I'm wondering if the original data injection issue that this thread was opened about may have happened in other studies too? I only found these bugged ids when doing a full Forest run on the Yale study.

[biblicabeebli]

I would say no, don't worry about it, just let me know if anything comes up. Forest should not be updated to handle this malformed data.

The cause was transient.

• There was a ~36ish-maybe hour period where - "uploading data from an hour for which we already had some data" - would cause those columns/lines to get injected/duplicated.
• After I fixed it I had to make a guess about how far back in "time as defined as the timestamp of the data point" I had to go in order to hopefully clear out all malformed data.
• This is not deterministic because a device can be off in the wilderness for a week and then upload a bunch of data from the past week all at once.
• All data was affected on all servers and so far we have this report of it.

This was your github issue, so you can make the call on closing it, I don't have anything else for you.

[hydawo]

This looks good. TLDR: there might be one or two users who had data in the small window of time that this error was occurring and had not uploaded it prior to Eli applying the fix above. If we encounter this again, Eli can fix the specific user/time period again.

Closed.

[jprince127]

Hello!

I wanted to add to this thread and say that I am also seeing a few participants with this exact data error for a different study. However, it occurs in both accelerometer and gps data.
@biblicabeebli would you be able to clear the affected user's data like you did with Max's study? The errors begin on 3/22:

study
IDs: u895ifsz, cmzge55q

@hydawo
Thank you so much!

[biblicabeebli]

@jprince127 responded in email thread, could you please close this thread when we the process is confirmed finished and fixed