ODKAggregate get submission list for forms

Hello all,
Wish that you all doing good!!
I'm here to ask question about the Briefcase API.
I'm using this document to do what I want : https://github.com/opendatakit/opendatakit/wiki/Briefcase-Aggregate-API

Now, let me explain what I'm trying to do:

I have a Web application that is build with PHP. The need is to grab data from Aggregate and make them available via the Web Application without [exporting from Aggregate, with briefcase and after import to the database's app].

So I need to quickly grab the data using the briefcase API.
Based on the link provided above and the ODK briefcase code. I understand that I can realise those step:

  • List the forms with the: /xformsList
  • Get the definition with the: /formXml?formId=
  • Get the list of ids of the submitted form: view/submissionList?numEntries=100&cursor=&formId=MEooo
  • And after download each submission using : view/downloadSubmission

My problem right now is that when I'm trying to get the list of ids of the submitted forms ODK Aggregate redirect me to the logging screen of ODK Aggregate that ask me my credentials

So I need to know or to get those IDs

Thank you in advance for your help

Hello all!!
I'm aswering my own question.

ODK Aggregate redirected me to the Log in page, because my request miss the Authorization header.
OpenRosa is based on the RFC2617 Digest Authentication.

This link https://docs.opendatakit.org/openrosa-authentication/ explain how to implement it.

So right now I can list all the instance of a Form with this Url : /view/submissionList
And also download a dedicated answer for a Form with this Url: /view/downloadSubmission

The problem for now is dealing with forms who have version.
I can download instances for forms who don't have version.
But I can't for the other forms

The error that I receive is:
HTTP Status 500 - top level group name: topElement is not as expected:

My formId string is as follow:
my_form_id[@version='3' and @uiVersion=null]/data[@key=uuid:3b475e55-c872-462b-b662-5d95e2d930cd]

If you have any idea please give me your help

Thank you!!

I'd recommend you step through the code at https://github.com/opendatakit/briefcase/blob/master/src/org/opendatakit/briefcase/util/ServerFetcher.java and see what we do there.

1 Like

Thank you @yanokwa

Right now I'm programming the logic in PHP in a way to export submissions as JSON.
This can be requested like a REST API.

Will share my code as soon as possible.

Thank you!

Hello all,
I have a problem right now.
It's about dealing with resumptionCursor.
When I send it to the /view/submissionList URL, the server returns an empty idList.

By making a comparaison between aggregate, I saw that the returned resumptionCursor isn't effectivelly the last one.

Here is my GET query to the endpoint:

http://my_uri.com/ODKAggregate/view/submissionList?numEntries=100&cursor=<cursor xmlns="http://www.opendatakit.org/cursor"><attributeName>_LAST_UPDATE_DATE</attributeName><attributeValue>2018-12-10T21:01:36.000+0000</attributeValue><uriLastReturnedValue>uuid:edf516b3-a17d-4c65-ba7b-71ca6868d1f9</uriLastReturnedValue><isForwardCursor>true</isForwardCursor></cursor>

&formId=ME18_Enquete_Structure_CS_v13

the Cursor value is
> >

> <cursor xmlns="http://www.opendatakit.org/cursor"> 
>         <attributeName>_LAST_UPDATE_DATE</attributeName><attributeValue>2018-12- 
>          10T21:01:36.000+0000</attributeValue><uriLastReturnedValue>uuid:edf516b3-a17d-4c65-ba7b- 
>           71ca6868d1f9</uriLastReturnedValue><isForwardCursor>true</isForwardCursor></cursor>

Hi, @rpalexis!

I'm a bit confused. Both cursors you have attached are equal. Is the second the one the server is returning with an empty id list? If so, everything is looking OK. Can you give more details?

Hi @ggalmazor
I'll try to explain the case.

  1. I have send a request with a cursor empty. So the server return me the cursor (That you call the second one)
  2. I send a request again with the value of the cursor( That you call the second one). A this step, the server returns the same cursor I send. With an empty idList.

The fact is that When I check the form on ODK Aggregate, there is more than 100 instances.

So, my requests tell that I gave only 100 instances but on Aggregate there are more than 100.

Hi, @rpalexis!

Could you try to export the submissions with Aggregate and see how many lines are in the exported csv?

Hi @ggalmazor!
Yes I did. The fact is that there are more than 100 submissions for the form.
When I'm querying the Aggregate API:

  • I found the first 100 submissions without specifying the cursor.
  • When I try to get the next 100 submissions "THE Agggregate API" returns an empty idList with the same resumptionCursor that I sent to the API

So it seems like I can't query thought each 100 submissions.

Hi, @rpalexis!

Have you tried to pull the form with Briefcase? If so, do you still get just 100 subs?

Hello @ggalmazor,
It's been a while that I did'nt work on this task.
I'm still getting stuck with using the resumptionCursor.

Is there a special encodage to use in order to pass the cursor parameter's value.

While the cursor is blank it returns the first one hundred submissions. That is Ok.
But the form has more than one hundred submissions.

I have tried the following:

  • Changing the xml's tag in html entities, this returns ids but the ids are the same one returned while passing empty cursor. [I'm using postman to do the test]

  • Encoding the cursor as uri encode

How can I pass it the Right way?

I was trying to read ServerFetcher.java' s file but can't understand well how the webCursorString is converted.
And passed to the GET request.

Thank you for your help

I made a PHP extractor to JSON.
Maybe you may see where a get stuck

Here is the link

@rpalexis I had this same issue. Solved it by ensuring the _LAST_UPDATE_DATE attribute is converted to UTC and formatted as YYYY-mm-ddTHH:MM:SSZ before sending it in the cursor param.

Based on the code at: https://github.com/getodk/briefcase/blob/master/src/org/opendatakit/briefcase/pull/aggregate/Cursor.java it is currently the only attribute supported to help retrieve the next results (in your case 101 - 200)

2 Likes