Pulling Unfinalizable Forms from ODKCollect with ODKBriefcase


(Faith Washburn) #1

What is the problem? Please be detailed.
I unfortunately had set up an ODKCollect form without properly assigning an RSA key, and we filled out about 90 forms before fixing the issue, which could not be retroactively applied to these 90 forms. Now I am attempting to pull these unfinalized forms from an Android device using ODKBriefcase. In the documentation, I see instructions for how to pull finalized forms from ODKCollect, but not for how to pull unfinalized forms. How could I go about this?

What ODK tool and version are you using?
I am using ODKCollect version 1.18.1.

Anything else we should know or have? If you have a test form or screenshots or logs, attach here.
I may just be able to use the same instructions for the unfinalized forms, but I don't want to jump in blindly and accidentally cause them to be corrupted (which is why I'm coming to you all first).


(Yaw Anokwa) #2

I believe Briefcase will pull all forms, even unfinalized/incomplete ones. For maximum safety, here's what I'd do.

  1. Reboot your device.
    • Makes sure Android has written every file to disk and doesn't have weird caching behavior.
  2. On the device, zip up, and transfer to the computer running Briefcase.
    • Makes sure Android isn't messing with the SD card and we also bypass Android's buggy media drivers.
  3. Create a new Briefcase Storage folder on your computer.
    • Makes sure we don't pollute whatever existing data you've put in Briefcase. And if things go wrong, you can throw away that storage.
  4. Pull once from the unzipped odk and then do an export.
    • Multiple pulls will cause duplicates on export.

Try the above and let us know!


(Faith Washburn) #3

Hmm, unfortunately I'm getting an unexpected error:

So it's saying that the form is encrypted and that I need to configure a PEM file, but that's not the case, as the forms were never even finalized due to their lack of encryption capability. (I attempted running it with my Private PEM key that I set up for the newer version of the form, but that didn't work, unsurprisingly.)

Your thoughts? I'm going to try to figure out a workaround but I wonder if this is the end of the road for these forms.


(Yaw Anokwa) #4

Hmm indeed. If you don't mind me seeing the data, please send me the odk folder at yanokwa@nafundi.com and I can take a quick look.


(Yaw Anokwa) #5

I got the export working and so I wanted to detail what I did once I got the zipped ODK folder.

Edit blank forms to remove encryption information
I looked at odk/forms folder to check out the form source. Both forms that were there had the <submission method="form-data-post" base64RsaPublicKey="blah"/> line that tripped up Briefcase, so I removed that line.

Remove submission.xml files
I looked through odk/instances folder in the zip that I received and there were some submission.xml files in addition to the normal instance files (e.g., My_Form_2018-10-19_16-10-36.xml). This is likely because Collect renamed the normal instance files to submission.xml to prepare for the encryption and the encryption failed.

I thought that might be a problem for Briefcase so I spot-checked a few of the submission.xml and confirmed they had the same data as the instance files and then I removed them. I'm not sure what would happen if you left them. Maybe duplicate rows in the export?

Anyway, if you are on macOS and comfortable with the command line, this command should work: find odk/instances -name 'submission.xml' -delete.

Export one form at a time
I noticed that the two forms had the same title and I happen to know Briefcase doesn't love that, so I imported the first form and exported to CSV and renamed the CSV. Then I removed that form data from the forms folder in the Briefcase storage directory and then imported the second form and exported that to CSV.

If you don't do the above, then the data from both forms will be written to the same CSV and since the headings aren't the same, it'll be a mess.