Integration of Enketo into ODK Central

(yes, i entirely ignored the installation questions for that response. i haven't had any good ideas yet.)

It's a configuration item. See here. Sorry, we are still working on creating more comprehensive documentation for Enketo (same project that funds this work).

I think you're misunderstanding the authentication requirements for the data collector that gets told to use a particular Enketo webform URL to start collecting data. Enketo has no authentication requirements itself for that scenario (and it stores no data). Form access and submission access are only protected by KoBo/Central/Ona/Aggregate if the forms are not public. Whenever Enketo receives a 401 response from Central/KoBo/Ona/Aggregate, when requesting the XForm or form media or external data resources, it redirects or collects credentials (depending on which auth mechanism it is configured to use). (Note that when it uses OpenRosa authentication those credentials are only stored in the user's browser). There is nothing new to be invented here, I think.

The single sign-on experience is supported for the second and third mechanism, but note that this may only be useful for the form administrator, since that is probably the only person that accesses the Central/KoBo/Ona/Aggregate interface.

1 Like

I suspect the date is postponed, isn't it?

Yes, I suspect so as well. We did get an extension from the donor, so that's good.

FYI I think this proposed new feature - ie integrating Enketo with Central - probably passes the bar as to what should be on the ODK roadmap. Perhaps someone on the TSC should add something to the roadmap for it? Perhaps someone on the TSC who also happens to be intimately involved in Enketo... (hint, hint :grin: )

2 Likes

so for what it's worth, we do have these items on the odk central roadmap already. i don't know enough to speak to TSC things.

1 Like

Thanks, good to know. Might still be useful to expose such a significantly/useful new feature on the (more visible?) ODK roadmap, but I'll let @martijnr make that call.

2 Likes

Wow..this went from August 2019 to June 2020.
Is there a specific reason why this was so much postponed?
I see that the integration of Enketo into Central is a key component for many people to switch to Central.
There technical problems or time constraints?
Is it not possible to anticipate this integration?

3 Likes

The Central roadmap is the projected plan assuming @issa and @Matthew_White are the only implementers. If @jnm, @martijnr and/or others are involved, that certainly shifts the timeline. I think we're all motivated to help make this happen as soon as possible.

I agree that it makes sense for these kinds of integration features to be reflected on the ODK roadmap in some way, if just as a table of contents that links to conversations like this one and Central/Enketo roadmap items.

I'm coming back to this after being out for a bit, has any discussion happened anywhere else? @issa, @jnm, any reactions to @martijnr's clarifications ? Is it time to schedule a call for interested parties to make sure all challenges have been identified and that it's clear who is thinking about what?

4 Likes

FYI as promised on TSC call, I've added an item to ODK roadmap to track.

4 Likes

any reactions to @martijnr's clarifications ? Is it time to schedule a call for interested parties to make sure all challenges have been identified and that it's clear who is thinking about what?

not from me. i don't feel like i have any further insight to add until i see some kind of a preliminary plan.

2 Likes

Hello again everyone, and thanks for all of your detailed and enthusiastic responses. My thought with prioritizing "Offline-capable, multi-submission surveys done by enumerators" as the first thing to tackle is that it would actually require minimal work (and hopefully be broadly useful). @issa, I'll outline my understanding below—please let me know if I make any missteps.

ODK Central is already an OpenRosa server supporting one OpenRosa client (ODK Collect), and my approach would be to treat Enketo as simply another OpenRosa client. Where Central seems to differ from other OpenRosa servers is that usernames and passwords are not used at all; instead, each "App User" authenticates with a unique URL made hard-to-guess by containing a 64-byte token.

Just as ODK Collect does, Enketo could interact with unique formList and submission URLs provided by ODK Central for each App User, forgoing usernames and passwords. Enketo would, in turn, generate its own unique, token-containing URLs: one per form per App User. For example, assuming all the forms are in the same project (ID 1):

Form App User Central OpenRosa URL Prefix Enketo URL
Well Pumps Meredith https://central/v1/key/{meredith-random-key}/projects/1/ https://enketo/::{meredith-well-pumps-random-key}
Well Pumps Jorge https://central/v1/key/{jorge-random-key}/projects/1/ https://enketo/::{jorge-well-pumps-random-key}
Well Pumps Ricki https://central/v1/key/{ricki-random-key}/projects/1/ https://enketo/::{ricki-well-pumps-random-key}
Cisterns Ricki https://central/v1/key/{ricki-random-key}/projects/1/ https://enketo/::{ricki-cisterns-random-key}

The workflow would be something like this:

  1. The system administrator of the ODK Central instance configures an Enketo URL and API key for use by the entire instance;
  2. [UX help, please!] ODK Central provides a UI element to retrieve an Enketo data-collection URL. This would be similar to "See code" in the "Configure Client" column of the App Users list, but there must be a way to select a particular form in addition to an App User;
  3. Once an App User and form have been selected, ODK Central POSTs that App User's token-containing OpenRosa URL and the <formID> of the form to the appropriate Enketo endpoint;
  4. Enketo stores the form and the OpenRosa URL in its Redis database, associating them with a unique key;
  5. Enketo returns a URL containing that unique key;
  6. ODK Central displays this URL;
  7. The project manager communicates that URL to the enumerator;
  8. The enumerator uses this URL to enter data and submits an instance;
  9. Enketo receives the submission and forwards it to Central's OpenRosa submission endpoint for this particular enumerator, which was stored in Enketo's Redis DB;
  10. Central recognizes the OpenRosa URL as containing a valid App User token and accepts the submission.

To respond to a few specific questions:

I propose that (at first) ODK Central does not ship Enketo at all. Instead, use of Enketo would be entirely optional, and the administrator of a Central instance could use any Enketo server reachable via HTTPS. Enketo already runs well in a Docker container, so savvy folks could set up Enketo immediately alongside their Central instance. If later there's a desire to include Enketo along with Central in a more streamlined way, I could help with the Docker aspect of that.

Central trusts that it can send blank forms and App User tokens to a particular Enketo that's been explicitly configured by the Central administrator. Central requires the Enketo server to use HTTPS with a certificate from a trusted authority to frustrate MITM and eavesdropping attempts. Central allows an instance to be submitted when it's POSTed to a valid, token-containing OpenRosa submission URL, no matter whether the client is Enketo or ODK Collect.

This sounds like it could be fun, but I'd like to avoid relying on anything but HTTPS over public interfaces. I may want to run my own Central but connect it to someone else's Enketo, or I might need a distributed cluster of Centrals and Enketos using something other than Docker's swarm mode.

Thanks for reading :slight_smile:

3 Likes

From a Central standpoint, the goal is to make Enketo functionality (preview, filling, editing) available with as little additional technical know-how or additional configuration as possible.

As far as code-paths, we can ensure there is not too much difference between bundled vs external if we're careful. After some discussion with @martijnr, it looks like bundled would use an Enketo subpath proxied to the local server in a Docker container. External servers would connect directly over HTTPS.

It’s riskier to bundle Enketo as a starting point and we’ve historically done the riskier stories first. We also aim to provide the best user experience first and fill in advanced features later. That said, I understand that starting with external access might make more sense for @jnm's roadmap/timeline.

I think we should get on a call to talk about the tradeoffs and dig into the proposed workflow. @issa, @LN, and I are available at the following times. @jnm, @martijnr, @Matthew_White which of these times would you be available?

  • Tue, Sep 10, 11 AM - 12 PM PDT
  • Tue, Sep 10, 1 PM - 2 PM PDT
  • Wed, Sep 11, 10 AM - 11 AM PDT
  • Wed, Sep 11, 1 PM - 2 PM PDT

0 voters

5 Likes

I've sent out a calendaring invite to the relevant parties for a call on Tuesday the 10th at 11 am PDT. The call will held at https://www.uberconference.com/opendatakit.

Everyone is welcome and if you are interested and can't make it, no worries, we'll take notes at https://docs.google.com/document/d/16btMIg7fz7rKURz5sZQPrDA83jHkEJXGOlyMvlPOrh8 and the audio will be recorded.

3 Likes

@yanokwa @issa @LN Any updates on this? Reading through the thread here, it's not entirely clear to me how you ended up wanting to do the Enketo integration and whether it is still on the roadmap.

Hi @Johannes_Wilm, welcome to the ODK Community, don't hesitate to take some time to introduce yourself here

this post mention that enketo integration into central will be done after june 2020 (july would be great :joy: :wink: )

Right, but that looks like it is from May 2019 and then it was updated, and then after that there seems to have been some protest here. And then the posts after that suggest that there has been further discussion on it - among other things a call in September 2019. And then it's unclear to me what happened.

I have done that now: here.

@mathieubossaert Question: Is it correctly understood that already now, using the OpenRosa API, it is possible to connect the Enketo frontend to ODK Central by installing the Enketo backend component? So this is just a question of bundling it together and and not needing two backends?

I'm sorry but I don't have any idea about that. I am just a enthusiast user of ODK tools, not a developper :slight_smile:

1 Like

It's an issue on the roadmap: https://github.com/opendatakit/roadmap/issues/37 (but that issue card won't be helpful since it just links back here).

We're aware that a lot of people could use an Enketo/Central integration. And would be made very happy by it. Unfortunately there are a number of things that the team is looking to complete first. You can check out the things being worked on currently in the Central block 8 release criteria

The Enketo integration is definitely still part of the plan; I think it happening after June is probably the current best bet (unless more people and resources emerge to push it forward sooner).

Thanks for your patience and interest in ODK @Johannes_Wilm!

4 Likes