Uploading to ODK Aggregate from non-ODK app

odk-aggregate

(Adrian) #1

Hello,

My company uses ODK Collect for data collection, however we have a certain type of data that can't be easily collected using ODK. I've developed an app with a more suitable UI and I'd like to be able to upload its data to our Aggregate server. The thinking is that I'll create a form with the appropriate data structure, load it onto our Aggregate server, then in the app generate an XML file for each piece of recorded data.

I'm new to Android development so I've been go through the source code of ODK Collect in the hopes of reverse engineering how it handles uploading to Aggregate. I seem to have run into a brick wall however, no matter what I do my connection to the server is refused. I figure it must be my code because the login settings are identical to what would be supplied in ODK Collect. Would someone mind looking over the code in question and giving me a few pointers? PasteBin below:

https://pastebin.com/mFEx6t2X

Thanks!


(Dr. Gareth S. Bestor) #2

Collect adheres to the OpenRosa spec (or here) regarding submitting forms, which you can implement using any suitable HTTP client (eg curl), passing the necessary HTTP headers, multipart POST data, etc. I'd start by looking at this spec to figuring out how to submit forms outside of Collect (rather than trying to decipher from the source code...). You might also want to look at the Briefcase API, which exploits portions of the OpenRosa.

Post back if you have any problems - I can probably dig up some curl code to share that does a submit, which should be easy to mimic in your app.


(Adrian) #3

I did look through the OpenRosa specs and attempted both a digest and basic authorization - both still shot me a permission denied error. I don't think my issue has anything to do with the headers, as the only required headers are date and OpenRosa version, which I supply in my code (see pastebin). I'm just stumped.


(Yaw Anokwa) #4

ODK Aggregate form upload via commandline (curl) has some examples that you can try from the CLI to make sure nothing weird is happening with your server.


(Dr. Gareth S. Bestor) #5

Ha. I completely forgot I posted that a while back :slight_smile:


(Adrian) #6

Hey guys,

Thanks a lot for your help. Turns out I forgot to add internet permission to the app (idiot), so I've progressed past that part. I'm now getting a 200 response back from the server when I attempt to post my file.

The file itself is not the issue - I decided to circumvent that problem by creating it ODK. Here is stream of the post headers/entity:

X-OpenRosa-Version: 1.0 Date: Fri, 21 Dec 2018 11:04:02 GMT+00:00 Content-Type: multipart/form-data --ngaTvCbJgSBQMNSTflV_McmMf0n2_NrH
Content-Disposition: form-data; name="xml_submission_file"
Content-Type: text/xml
Content-Transfer-Encoding: binary

(Form content omitted)

--ngaTvCbJgSBQMNSTflV_McmMf0n2_NrH--

I'm posting to ODKAggregate/submission and when I try to manually upload the form from Aggregate, it works fine. The headers look good to me, any idea what I'm missing?


(Yaw Anokwa) #7

My recommendation is to look at the Collect and Briefcase code and use that. If that doesn't help, use HTTP proxy (I like Charles) to see what your app is doing.


(Adrian) #8

So I took your advice and used Charles - turns out I was receiving a 401 authentication error on the POST but my app was relaying the 200 on the GET statement. (Which makes zero sense to me).

The authentication error said that the server required full authentication and all I could figure was that I needed to use digest authentication instead of basic. I rewrote my code to do so and I'm still getting a 401 - this time it says "This request requires HTTP authentication (Incorrect response)".

I went ahead and submitted a form manually through the Aggregate server in Firefox and pulled it's POST statement as a point of reference. It is nearly identical to what my app is sending so I have no idea why it is failing. Is it an encoding issue?

Charles POST:
https://pastebin.com/5vGwGip3

Firefox POST:
https://pastebin.com/MnyVGhK6

Updated Code:
https://pastebin.com/RKWzUgR4


(Dr. Gareth S. Bestor) #9

Have you read thru all is this: https://docs.opendatakit.org/openrosa-authentication/


(Adrian) #10

Yep. My header is consistent with what the documentation stipulates. For example, here is their sample of a valid request header:

Authorization: Digest username="Mufasa",
realm="testrealm@host.com",
nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093",
uri="/dir/index.html",
qop=auth,
nc=00000001,
cnonce="0a4f113b",
response="6629fae49393a05397450978507c4ef1",
opaque="5ccc069c403ebaf9f0171e9517f40e41"

And here is what mine produces:

Authorization: Digest username="user",
realm="Data Collection ODK Aggregate", nonce="MTU0NzYwNTQ4NjgwMjo5ODU3NTJiOWVkMTBmMzM1MzhhYTBjNjAxNjFlOTVkMQ==",
uri="/ODKAggregate/submission",
qop=auth,
nc=1,
cnonce="a39ae04a",
response="b725ffa3b9e758cc90b01f8ca4f8ccf5"

The only differences I see is that my nc is programmatically set and opaque is omitted simply because the server doesn't respond to my original GET with that data. The documentation says that the server may not always send opaque and if I understand the request counter correctly, 1 should be fine since each time the code runs it requests a new nonce.

Unless I missed something in my read through, this should be fine.