What do you do with temporal data from your submissions?


(Guillermo) #1

Hi ODK!

The ODK 1 TSC is studying how temporal data (dates, times, dates with times) is used by end-users like you! :spiral_calendar::alarm_clock:

We're interested in knowing what happens to temporal data in your forms after exporting submissions to CSV files or publishing them to Google Sheets and Fusion Tables. We want to know, are they used just for ordering rows? are they transformed to calculate durations between events? maybe other fancy uses?

Also, what are the most frequent hurdles of working with temporal data collected with ODK? Want to share any horror story about time zones? Know any neat tricks to make them compatible with third-party apps (Excel, LibreOffice, ...)?

If any of your forms has date, time or dateTime fields in them odds are that sharing your experience and thoughts is going to help us!. Write a comment and let us know :wink:


(Jason Ives) #2

For our largest current project, we use timestamps to monitor how long interviewers are taking to complete sections of the questionnaire, as well as start to finish. Average times are calculated and large deviations from that average (especially entries much faster than average) are investigated for extenuating circumstances, fraudulent collection, etc. I believe this information is also used to inform survey design changes for future data collections.

I can't speak to analysis or data management issues with these fields, although I believe any work done with these is in SAS or other statistical software.

Jason


(Richard Fletcher) #3

I have used timestamps (as metadata rather than a direct response) for general monitoring as I'm sure many/most people do.

There was one form where I used date input and then compared these to a separate date. Eg: someone booked for an event and we wanted to know how far in advance they had booked. There were around 30 different event dates and at least 500 bookings. I think I converted the dates into numeric to make the calculation in excel or Google sheets.

The form being used was in enketo, for this usage it was handy that the date input defaulted to 'today' as the aim was for people to use the form immediately after a booking but in some cases they did it after the fact.


(James) #4

I've used dates and times both for auditing purposes (hidden fields) and as a means of recording events. On the back end, we use it to search for transactions that occurred within a date range or to filter things. E.g. "in which growing season did this occur?" We use math to e.g. count days between events. E.g. "How many days between planting the seed and when the plants flower?"

There are a few tricky bits about dates:

  1. You can't depend on them: Time, especially as reported from mobiles, is notoriously unreliable. Users set dates wrong, cell towers report and sync time wrong, etc. I've seen log files where time literally counts backwards and skips around on a properly functioning mobile (with conflicting time authorities and a bad cell tower).
  2. There are times when you want to make a field mandatory (e.g. someone's birthdate for recording their age for M&E), yet there are times when the date legitimately isn't know. E.g. if someone's birthdate is unknown by them and the community, what do you enter? In Tanzania, the most common official birthday is July 1, half way through the year, the default when someone doesn't know their real day of birth. It's nice having an "unknown" option that can be selected, even for a mandatory date/time field.
  3. Date and time formats can be tricky when routing through other tools like spreadsheets or other countries that use different representations.

(Guillermo) #5

Thank you all for your answers!

Keep'em coming :slight_smile:


(Jeff Davids) #6

At SmartPhones4Water, datetime is a mandatory part of every ODK form we use. Once the data is in aggregate, we use a Z-Alpha JSON publisher to get our data into MongoDB. We have a custom Python app that processes these data. Several reports use datetime as a way to develop aggregate statistics, and develop time series of data. For example, we use datetime to determine how many measurements different citizen scientists are taking per week, month, etc. We use datetime to develop time series of water level measurements. For precipitation, we use datetime to determine the amount of time elapsed from the previous to the current record, in order to calculate an average rainfall rate. In summary, datetimes are foundational to everything we do, and I would expand this to say that anyone in the geophysical sciences would probably agree. We have been really happy with the consistency and reliability of ODK datetimes (and everything else the community supports). Keep up the good work.