What's coming in Central over the next few years

The Technical Steering Committee that guides the ODK suite has shared in our upcoming changes to Aggregate topic. Following these changes we are slowing development on Aggregate in order to concentrate on Central. We expect Central to become a viable replacement for Aggregate for most users by June 2020.

As we make progress towards that goal, we wanted to provide more visibility into our roadmap and timeline so you can offer feedback. We are particularly interested in the features Aggregate has that you need to switch to Central.

Please note that our goal for Central isn’t to build an exact copy of Aggregate. Rather, we want to make a new ODK server that works better for most users and is easier for our community to evolve.

As always, we’ll be able to move faster if we have your help. If you or someone you know can fund this work or help with the development, please email yanokwa@nafundi.com.

Features we support now

  • User accounts and management
  • Role-based user permissions
  • Projects to organize users, permissions, and forms
  • Form and submission upload and management
    • With form and submission multimedia or data attachments
    • With a table preview of submission data
    • OData live data feed for analysis with tools like Excel, Power BI, and Tableau
  • Integrated checklist-based help system
  • Automatic, encrypted off-site data backups to Google Drive
  • Clean and modern REST API for integration and extensibility
  • High performance on low-cost hardware or cloud providers
  • ODK Briefcase-compatible data output
  • Encrypted forms (self-supplied or project managed keys)
  • Briefcase push/pull support

You can find out more about these features at https://docs.opendatakit.org/central-intro.

Features we’ll likely add before June 2020

  • More fine-grained permissions for projects
  • Sync the device to some centrally managed state
    • We’re aiming for a single button push that will sync forms, submissions, and settings.
  • View all submission data
    • We currently let you see the first 10 fields in a submission. We’ll soon let you specify a limited number of fields (~30) you’d like to see in the user interface.
    • We’ll also enable viewing of data inside repeats and showing missing media submissions.
  • Publishing to JSON
    • We have an OData feed and plan to add some kind of JSON-basd webhook.
    • You will soon be able to push/pull/export with Briefcase.
  • Export to CSV
    • We currently support CSV exports (on the server and through Briefcase).
  • Documenting running on other Cloud providers
    • We’ve provided documentation on Digital Ocean, but will add AWS, GCloud, and Azure.
  • Upload XLSForms in addition to XForms
  • View and download form XML/XLS and attachments
  • Filtering submissions by submission metadata (e.g., datetime received on server)

Features we’ll likely add after June 2020

  • Form versioning and updates
  • Longitudinal data collection
  • Preview and fill out forms using Enketo (in partnership with @martijnr)
  • Data editing/cleaning using Enketo (in partnership with @martijnr)
  • Delete all or some submissions, true deletion of forms
  • Filtering submissions by any field in the form (likely not real-time)
  • Bulk upload-download user email address and permissions
  • Improved install and upgrades to reduce dependence on command line

Features we’ll likely never add

  • Parsing submissions into database tables
    • Support for arbitrary schemas is a hard problem that introduces a lot of complexity (see Aggregate’s codebase).
  • Visualizing data as pie or bar chart or map
    • We believe this functionality should be pushed to external services through OData
  • Arbitrary database support (e.g., MySQL, SQLServer)
    • We believe PostgreSQL is the best database for our workload.
  • List of previously exported submissions
    • This shouldn’t be necessary since you can regenerate exports quickly
  • Toggling display of submission metadata
    • We will always show you all the relevant data.
  • Export to KML or some other format
    • We do not support KML, but GeoJSON is supported through Briefcase.
    • If there is a wildly popular export that users want, we’ll consider it, but we’d rather not add to our maintenance burden.
  • Publishing to Google Sheets or some other web service
    • The webhook should allow you to connect to other services that can then forward data on to Google Sheets.
    • If there is a wildly popular publisher that users want, we’ll consider it, but we’d rather not add to our maintenance burden.
  • Anonymous retrieval of images, audio, video for visualization
    • We expect downstream visualization tools to be able to securely auth against the server
  • Manually upload submission data
    • We believe this is a rare use-case and Briefcase supports it well.
  • Support of the Briefcase-Aggregate API
    • We believe the extensive RESTful API is better.
  • Running on Google App Engine
    • We value portability over the benefits of a platform as a service. You will be able to use Google Cloud, just not Google App Engine.
  • A non-Docker install
    • We use standard components, so it’s possible to set up Central outside of Docker. We don’t recommend it because we don’t have the resources to support non-Docker installs.

Some considerations to be aware of

If you use App Engine, there’s currently no way to redirect traffic from an appspot domain to another domain. You'll need a plan for changing the server URLs on your devices. If you use your own domain (e.g., with the Aggregate v2.0 or later), that domain can be used with Central.

Our expectation is that most users will keep using Aggregate for ongoing campaigns and will start new campaigns with Central. That is they won’t migrate their data. If migration is needed, Briefcase will soon be able to migrate blank forms and submissions from Aggregate to Central. Publishers, users, filters, etc will not be moved.

Our goal is to make Central fast, secure, and easy to use. Using cloud hosting is the best way for our small team to reach that goal. Unfortunately, cloud hosting these days isn’t free, so you should expect to pay $5-10/month. If $5/month isn’t affordable, then you can host on your hardware and we’ll provide some documentation, but we’ll be limited in the support we can provide. Contributions here are very welcome.

This timeline is not set in stone! We’ll continue to update this post and the timelines as we get new information, so please watch this topic to get updates.

I have a question or comment

Put it below :point_down:

2 Likes

Should these be captured in Server Logs though? So for example I as an admin can know that @chrissyhroberts as a project manager downloaded the data at a certain date-time?

1 Like

@dr_michaelmarks presently this is not planned.

but i know for hipaa / similar reasons it can be a requirement so i'm not opposed to adding it someday.

1 Like