Odk collect is taking too long to save survey

odk-collect

(Alexandre A De Oliveira) #1

What is the problem? Please be detailed.

Hi ODK community,
I am using odk form, built in xls and using xls2xform to export it as xml. Our data collecting is about trees in a tropical forest and there is several particularities that we implemented in xlsform:

  1. populated form with previous data (old census) using a .csv file
  2. svg figure to select subplots (operational areas) as a question, selecting svg id.
  3. svg figure for each subplots with previous mapped trees used as question to select tree (svg id) and confirm mapping
  4. svg figure for mapping new trees
  5. repeated questions for subplots, trees ans stems (trees with more than one stem)
    6 repeated question for trees that where mapped in previous census but not found , at the end of each subplot to make sure the field team look for it
  6. All processes is done offline: load forms into ODK collect, exporting (ODK briefcase), merge data, storage an analysing

Despite this particularities, i managed to build the form and is working. During a field working day the team are able to do one plot (16 subplots) with 200 trees, or even more. The problem is that when we try to save the form, after a working day, odk collect is taking too long to save. Several hours even days for saving just one day of work. There is no error message or warning, only keeping showing the message "validating answers" (or something like it, we are using portuguese language form!).

What ODK tool and version are you using? And on what device and operating system version?

The last survey data spent more than one day to save built a 300 KiB xml file in the Instance folder inside odk device directory . The device is a samsung SM-T385M, one of the best one that one can get in Brazil market, and we do not have memory problems. The ODK collect is version v1.18.1

Anything else we should know or have? If you have a test form or screenshots or logs, attach here.

The below link has our form (xls, xml and media). Media for two plots A00 and A01 (to able to see maps during the survey one should select column "A" and line "00" or "01"at the beginning of the form).

I have spend several months working to build the odk collect form and to trainnning team to use it, and now we are loosing data and time. I do not want to go back to paper, some one could help and take a look to see if there is any wrong in the form? Should I give up to use some of the functionalities to make form more efficient to save? Any suggestions is wellcome!

https://drive.google.com/open?id=1vB1LUdvEj2U0jP6Zk32CST3TLP21EEeP


(Yaw Anokwa) #2

@aleadalardo Can you do some testing to document the speed of saves for different scenarios? For example, are saves fast if you do 1 subplot with 1 tree? What about 10 subplots with 1 tree? What about 10 subplots with 10 trees?


(Alexandre A De Oliveira) #3

Hi @yanokwa, thank you for your answer and concern . We did multiple tests and yes the time consuming is larger for larger amount of data and if the number of sample units increase. Since we have 16 subplots (work units) compounding a plot and around 200 trees, our basic goal of field working day , we tested ways to decreasing time with this amount of data. We found that the size of the csv file with data from the last tree census it is important for time saving as well as the last repeated question "rep_miss". This last repeated group show to users the tags numbers of trees in the last census that was not found in the current one, showing map with trees positions and ask the user if there is any evidence of dead for this missing tree. For me who is curating and analysing data this is a very important step to make sure if the tree is dead or field team did not found it. This garantees that the team will look for it in the place that the previous team mapped it.
Since the complet form is taking more than full night long to save and reduce the previous data set is not an option, we are using a second form without the last repeated question. This is not the ideal solution but was the best we can do since there is 3 team in the field by now and we only have 3 tablets. Even with this reduction, the time to save complet form is around 3 hours.
I really appreciate any help from ODK team. I know our form is a is a little complicated. I spent a lot of time to understand the xls sintaxe in order to be able to get the best from odk collect. And I thougth I had reach my goals when I did the first tests with it. I was proud to got it done in time to start field work using it. It is frustating that odk collect have such basic restriction as this and I do not have knowledge of java or javarosa to help to fix it. It is a very nice tool and I appreciate the effort of so many people involved with its development. Unfortunatly, In our case ODK seems not to have been the best choice, since it can not be used as we plan in order to monitoring tree fate in tropical forest.


(Alexandre A De Oliveira) #4

IT IS NOT ONLY TAKING LONG TO SAVE, ODK COLLECT IS STOPPING SAVE PROCESS AFTER SOME TIME.