Integration of Enketo into ODK Central

They/he/she definitely will. Thank you!

@martijnr do you know how Enketo will track changes made to instance through the server?

With @ln and @yanokwa we are doing quite a bit of work trying to get a GCP compliant audit trail

At the moment we don't worry about server changes to data (because they aren't possible) but if this functionality became available being able to similarly track changes made via the server interface would be excellent and a huge plus on the GCP side.

1 Like

Hi to all, I understand from that discussion that you will make ODK Central able to serve collect forms as web forms. Am I wrong ?
If not this will be a really great new feature !

Mathieu

1 Like

First we'd have add the general audit capabilities to Enketo. I'm open to getting the audit features that ODK Collect has if we can ensure that (meta)data is compatible when collected by different clients (Collect & Enketo). I'm not sure if that is feasible for the complete package of audit events, but for value change logs I imagine it would be since it will be straightforward (not relying on when a field has focus as in the 'question' event). So Enketo may end up with a subset of audit capabilities that ODK Collect has (provided there's funding and developer interest).

Adding the ability to edit already submitted data via Central, could work with whatever audit capability gets added to Enketo, but we'd have to add something to Enketo's edit API endpoint to send the previous audit file with the API call. That is feasible. Once Enketo has the file, loading it would use the same mechanism as loading an unsubmitted draft record (with audit file) from the browser storage.

2 Likes

Yes, it will allow serving ODK XForms as web forms indeed! (like in Ona and Kobotoolbox)!

I'm a little late to the party but just described something similar at Approaches for linking form instance changes to individuals.

That seems totally fine to me. In fact, I imagine that Enketo might also want to introduce event types that don't make sense from a Collect perspective.

What we might want to do sooner than later is add a way for the client audit log to include a user agent identifier (ODK XForms client type, client version, operating system, etc).

1 Like

Yes, indeed! That would be a good to start early on.

1 Like

since this is a thread about Central after all, i thought i'd chime in and just say that somewhat fine-grained data history (both forms and submissions) and administrative action audit logging are factored into the architecture of Central. i don't think at the interface level it works exactly the way enketo et al do it, but after all it's just architectural foundation right now—exactly what happens upon what action can be decided upon later.

1 Like

Thanks @martijnr.
Such a lot of perspectives ! :smile:

1 Like

Hi everyone (and @issa in particular :smile:),

I work on KoBoToolbox (one of the partner organizations that @martijnr mentioned) and would like to get some of this done before the end of the month(!) Having read through this thread, it seems like Editing already submitted data probably needs further discussion and won't come together so quickly—but some of what's under the Viewing/pre-viewing/pdf-ing forms and Collecting data in a browser headings could probably be done before September.

Here's how I thought I'd sequence the work:

  1. Offline-capable, multi-submission surveys done by enumerators
  2. A form preview to test form appearance and logic, as used on https://opendatakit.org/xlsform/

…and, if there's still more time,

  1. Viewing a record in readonly form
  2. A PDF of an empty form
  3. A PDF of a record loaded into a form
  4. Online website-embedded surveys
  5. email single-use surveys that have some protection against multiple submissions by the same person

:star: What would be the best way to get started? There are a few design (UI and code) decisions to make; I could start another topic here or bring them up on the Slack.

Thank you!

5 Likes

hey @jnm, et al:

really happy to see some movement on this. :slight_smile:

i think before we proceed too far we need to work out a couple of things:

  1. what is the proposed method for shipping and operating enketo in our environment? is it optional or always present? how do we manage upgrades?
  2. what is the permissions and authentication methodology between the systems, both at the service level (central can trust this enketo because xyz is true) and at the user level (this instance can be submitted because xyz is true).

i think for these reasons and more, it probably makes the most sense to focus on read-only features first. previewing a form, both interactively and as pdf, and viewing existing single submissions. i think for enketo to actually submit anything we have to really sit down and puzzle through how these features are likely to be used and therefore what sort of permissions/user tracking models make the most sense, and i think those conversations will take some time.

i will warn you that i will be a little bit of a pain in your butt on user experience and on security. i do not hold absolute power over this project but that doesn't mean i can't be loud and annoying. :slight_smile: i would really love to see proposed design and criteria (see our own release criteria for v0.6 for an example of our process on this) before work proceeds. some sort of similar proposal process for the technical architecture will help this process a lot.

finally, i will suggest one direction for solving some of these problems: we can enable backchannel privileged communication between services within the docker-compose cluster, and central/enketo can mutually trust any requests made over that communication without authentication. this can be done by exposing a secondary API over a private port that isn't exposed outside the docker network. the asterisk here is that for the purposes of the server audit log (coming in v0.6) it will still be necessary to identify the actor invoking each action.

it may also make sense (or not) to avoid exposing enketo entirely and proxy everything through central.

4 Likes

Not that we need to wait till then, but this should make for a great discussion at the convening!

Right! But my concern would be that this would be too late to be able to use the funding we have now (knowing how busy @jnm gets ;)). Hopefully we can get this fleshed out beforehand.

Do you have a preference for a technical solution @issa? I do not (but also am not familiar with docker). Whatever is best for users to use, install, and update Enketo. It might be useful for scalability if the solution facilitates hosting Central and Enketo on different servers, but maybe that is too complex (for the target user group). I know an important consideration for Central is the cost of running it. It is not really recommended to run Enketo on a droplet with less than 2 Gb of memory ($10), so maybe that means it is best to make Enketo optional.

A. For Enketo to allow only API calls from the trusted Central/KoBo/Ona/Aggregate server, there are 2 levels:

  1. required: an API key provided in the Authorization header with type Basic as the username part (password part is empty)
  2. optional: the Enketo server can be set up to only allow a specific form server domain or domain + path (e.g. only accept the server_url API request parameter value 'https://opendatakit.appspot.com').

These API calls return a webform URL.

B. Submissions to, and obtaining form resources from Aggregate/KoBo/Ona/Central:

Enketo supports 3 methods: https://enketo.org/develop/auth/. I saw a while back that Central supports OpenRosa authentication with a small change (no handshake, if I remember correctly). That would be a minor change we could support in Enketo if (really) necessary. It would be great to use one of these 3 supported methods. KoBo uses the first method. Ona uses the second method.

1 Like

is there additional documentation on how the external authentication processes work? how does enketo know where the login page ought to be, what does it expect back, etc?

if i'm reading this flowchart correctly, can some of these redirection steps be bypassed if a token is already provided with the initial request?
central has a relatively robust temporary token system that can grant very fine-grained/limited permissions for time-limited tokens, so that could be used here; albeit it sucks to have to do database writes to perform cross-service read operations.

in general, i would be pretty disappointed if we can't work out some kind of relatively seamless/single-sign-on experience between central/enketo, where you never have to understand that you are authenticating to two separate services.
it seems to me that if we can't achieve a fairly tight amount of integration for the eventual end user, we may as well just make sure the services can play well near each other and provide documentation on how to install the two completely separately and then use them together, rather than put a lot of time into interoperation and co-installation only to have the end experience not reflect that investment.

(yes, i entirely ignored the installation questions for that response. i haven't had any good ideas yet.)

It's a configuration item. See here. Sorry, we are still working on creating more comprehensive documentation for Enketo (same project that funds this work).

I think you're misunderstanding the authentication requirements for the data collector that gets told to use a particular Enketo webform URL to start collecting data. Enketo has no authentication requirements itself for that scenario (and it stores no data). Form access and submission access are only protected by KoBo/Central/Ona/Aggregate if the forms are not public. Whenever Enketo receives a 401 response from Central/KoBo/Ona/Aggregate, when requesting the XForm or form media or external data resources, it redirects or collects credentials (depending on which auth mechanism it is configured to use). (Note that when it uses OpenRosa authentication those credentials are only stored in the user's browser). There is nothing new to be invented here, I think.

The single sign-on experience is supported for the second and third mechanism, but note that this may only be useful for the form administrator, since that is probably the only person that accesses the Central/KoBo/Ona/Aggregate interface.

1 Like

I suspect the date is postponed, isn't it?

Yes, I suspect so as well. We did get an extension from the donor, so that's good.

FYI I think this proposed new feature - ie integrating Enketo with Central - probably passes the bar as to what should be on the ODK roadmap. Perhaps someone on the TSC should add something to the roadmap for it? Perhaps someone on the TSC who also happens to be intimately involved in Enketo... (hint, hint :grin: )

2 Likes

so for what it's worth, we do have these items on the odk central roadmap already. i don't know enough to speak to TSC things.

1 Like