Have Collect exactly match the forms on Central

This is an initial step towards servers (e.g. Central) being able to manage more of the client (e.g. Collect) user experience. This feature description was developed with @issa from feedback given during Central user interviews.

1. What is the general goal of the feature?
Data collectors can end up with test or old forms on their devices. That can make it harder to find the correct form in Fill Blank Form. Similarly, data collectors may forget to download or update some of the forms that they’re supposed to fill out leading to incomplete data collection.

This feature would make it possible to instruct Collect to exactly match the forms that a project administrator has made available for the App User in Central.

With a server that has this feature turned on, when the data collector taps Get Blank Form, all forms would be selected and grayed out to indicate that their state can’t be changed. At the bottom of the screen, there would be a single Update button.

When the user taps the Update button:

  • all form definitions in the form list sent by Central would be downloaded or updated
  • all form definitions not in the form list sent by Central would become invisible in Fill Blank Form (soft delete)

Filled forms associated with soft deleted form definitions would still be available in Edit Saved Form, Send Finalized Form, and View Sent Form. Soft deleted form definitions and associated submissions would be visible from Delete Saved Form. Once all submissions associated with a soft deleted form are deleted, the corresponding form definition is also deleted.

If a form that was previously soft deleted is downloaded again, it will become visible in Fill Blank Form.

2. What are some example use cases for this feature?

  • During field testing for a new data collection project, a project manager publishes a new form that replaces a previously-used one. Currently, she personally updates every data collector’s device to make sure that obsolete forms are removed from the devices and that the correct versions of all forms are added. Using this proposed feature, data collectors can tap a single button to ensure that the forms on their device matches the ones that they should use.

  • A project manager has retired 3 forms that no longer need to be filled. Using this feature, data collectors can tap a single button to ensure that they won’t accidentally fill those forms out.

  • A data collector collects data for several organizations. On a day when she is working on Project A, she can log in as the App User for that project, tap a single button to get all the forms related to that project and clear out the ones she doesn’t need.

3. What can you contribute to making this feature a reality?
Project management, Collect implementation.

as the originator of this request, i suppose i should say something here, though @ln has done an excellent job describing the proposal and its motivation.

in general, if i were to characterize the greatest sources of pain and anxiety around practical ODK deployments that i have heard, very near the top of the list would be insecurity around what exactly is happening on devices in the field. as it pertains to forms and getting them onto devices:

  1. which forms are on each device?
  2. what version of each form is on each device? has the device been appropriately updated to the latest versions?
  3. are people still filling out forms that i've deprecated, or outdated versions of forms?

one side effect of these concerns is that in cases where these anxieties become major practical problems, a lot of deployments heavily centralize the management of their devices: enumerators are not trained or allowed to update their devices, and instead a small number of highly trained staff handle all form updates and management across all devices themselves.

but this approach carries its own new burdens; for one, this creates situations eg during training where staff are up all night spending hours upon hours manually updating each device for a form version change, and for another these sorts of tedious rote management tasks lead to frequent errors that are difficult to catch.

the root of the problem, as i see it, is that when strict centralized control is desired, it is not available. there are a number of possible solutions to the general problem of on-device form inventory and version updates, but as i see it the anxiety will not go away until there are mechanisms available for airtight, highly guaranteed centralized configuration of devices.

hence, this proposal.

4 Likes

Thanks for the examples and descriptions of the problem we're trying to solve here @LN and @issa! I have a couple of clarifying questions. Since they're about the problem space itself I thought I'd ask them here rather than in the discussion around the OpenRosa spec proposal:

  1. Do we think this feature will need to exist in clients other than Collect?
  2. Do we think project managers will need to enable this "mode" midway through a project?

I'd been thinking that we could introduce a Collect setting ("Match server forms exactly" for instance). We could implement in Collect first and then add a feature in Central that adds the setting into the QR code it provides for configuring Collect.

However, I have an inkling the answer to both questions is yes. If so, it does feel like we need some kind of server side "declaration" (like the one proposed) that the client should behave this way as:

  • it makes it optional for clients to implement while also encouraging them to do so - it'd be in API docs/specs
  • it allows project managers to change their minds without then having to reconfigure individual devices one by one

Does that make sense?

This is hard to answer because while we know that there are many compatible clients out there, the only ones we have much knowledge about are Enketo (@martijnr) and iXForms (@Xiphware). Enketo doesn't have a notion of "a set of forms currently on the device". However, we know that wrappers around Enketo do. My feeling is that if the feature is simple to implement, it's the kind of thing that clients that do maintain a local set of forms would do.

I don't think so. I think that conceptually it would be ok for this to be something that is set once. I do think it's important that this is dictated by the server, though. So the big question is when this "once" would be.

From a user perspective, I think it would be acceptable for this to be set when configuring the client with a barcode. But then we get back to your point 1 -- it feels like once we start putting more settings into the barcode there should be a cross-client specification for those settings so that all clients can benefit.

yeah, i agree with this. if i get my way this will be turned on by default for all Central projects, and if you don't want this behaviour the toggle to disable it is somewhere deep in an advanced settings page.

i do think getting to that ideal state might involve making (or pushing/forcing) progress on whatever concept we might land on for .. let's call them multiple Collect client profiles? i don't know what the current terminology for that pseudo-login concept is. client multitenancy.

one more thing i would like to add:

eventually, i think it would be nice to add a mechanism whereby some kind of easily distinguished identifier could be generated or attached to each "version" of the form inventory itself, which could be displayed on eg the Collect homescreen, so that it's really easy to tell if a device is out of date.

2 Likes

A big "yes" to this proposal, "yes" to "client multitenancy" (nice terminology), and "yes" to making it easy to tell if a device is out of date.

We have an ongoing polio-related project (not ODK, but data collection, kinda sorta), where the single biggest problem is with people accidentally using out of date hardware - or rather hardware with out of date software installed. Managing these things at scale is really hard, so anything that makes it easier gets a :+1: from me.