ODK Central v0.6 Beta

issa · August 15, 2019, 8:52pm

Central betas are an opportunity to get community feedback on ongoing work, so check out the introduction and follow the instructions to try it out on our sandbox or your own machine!

Note: if you are using Briefcase with Central, please be sure to upgrade Briefcase to at least 1.16.3 before using Central v0.6, due to minor changes in the authentication systems.

Server Audit Logs

New to Central v0.6 are Server Audit Logs. These track administrative actions performed on your Central server: new users, role changes, form creation and deletion, and more. You can access these under the System tab at the top of Central. We have already been logging some (but not all) of the audit log actions for a while now, so you may even see entries in there already.

To learn more about Server Audit Logs, including the list of actions we track, please take a look at this article.

Encryption

We now support Form Encryption. If you already use encryption with Aggregate today, you should be able to use exactly the same methodology with Central, including the use of Briefcase for data decryption and extraction.

But, Central also supports a new way to manage your encryption: Project Managed Encryption. For those of you who wanted to use encryption previously but found the procedure challenging, and for those of you who are unsure about how to secure the encryption keys you have to generate to use Aggregate's encryption, Project Managed Encryption is a far easier path to encrypted forms. You don't even have to use Briefcase!

With Project Managed Encryption, Central will generate and secure your encryption keys for you. They will be locked behind a passphrase that you give it: without the passphrase, the stored encryption keys are useless, and Central never stores your passphrase anywhere. To enable Project Managed Encryption, go to the Settings page on your Project and choose a secure passphrase. To retrieve your data, click on the Download data button as usual, and you will now be asked for your passphrase. You will get your data back in a zip file. That's all there is to it.

Encryption is definitely not for everybody. Central operates over HTTPS and so it should be secure for most purposes by default. If you turn encryption on, you will lose access to encrypted records over the OData feed, and if you use the old Aggregate self-supplied key method Central cannot help you decrypt your data. As of v0.6, once you enable encryption on a project you will not be able to turn it off again.

For more information and a step-by-step guide about form encryption in Central, please read this page.

Other Highlights

Thanks in part to feedback from our users and contributions from community developers, we have made the following improvements:

We have integrated instructions for using the free and popular R statistics application with Central OData, using the ruODK library and documentation created by @Florian_May.
Improvements to the OData feed: fixing some broken cases with weird field names (#199, reported by @stephen_k_ojwang), and adding accuracy information to geographic points (#213, suggestion by @Florian_May).
We now detect expected form attachments implied by simple uses of the search() appearance (#221, requested by @saif2khan).

As always, we have updated our User Documentation and Developer/API Documentation for the latest changes. More notes about changes to the API for v0.6 can be found in the Changelog, which lists additions, breaking changes, and other things of note for each major release.

Final Notes

For a longer term roadmap, read what's coming in Central over the next few years. For what's coming in our next release, see Central v0.7. We would love to hear your thoughts if you have any.

We encourage you to try Central and put your feedback in a comment below We are particularly interested in answers to the following questions.

What, if anything, was confusing?
What missing feature is preventing you from using this for your next project?

dr_michaelmarks · September 4, 2019, 6:41pm

@issa
A few minor things as I work them through on our test server at LSHTM
When you retire a form (closing/closed) it would be nice if this was indicated somewhere on the Project Overview and Form pages.

For example here on Overview page it is not at all obvious that these forms have been Closed

Same applies on the Initial Form view here:

Only if I click all the way through to the individual form settings can this be seen:

Some simple flagging of form status on earlier pages would be very helpful

issa · September 4, 2019, 7:34pm

hey @dr_michaelmarks, that's really great feedback. thanks! i think we have one item coming in v0.7 that will improve this situation a little, but i agree that surfacing the form state in more places would be really helpful.

dr_michaelmarks · September 4, 2019, 7:39pm

@issa next point

Our server address for our test is

XYZ.lshtm.ac.uk

If I try and add a webuser with a non lshtm.ac.uk email address I get:
"Something went wrong: the server returned an invalid error."

issa · September 4, 2019, 7:55pm

hmm. did you configure central to talk to an internal mail server of your own, and if so is it allowed to send outgoing mail?

dr_michaelmarks · September 4, 2019, 8:05pm

Copying @MatthewMac from out IT who set this up.

Related point is the link formatting in the email seems a bit broken

dr_michaelmarks · September 4, 2019, 8:29pm

Next UI comment @issa

It is very non-obvious what the Project ID is for a given project - relevant if you are using Briefcase to export (for example if using form level encryption).

Obviously it is in the URL but it is not actually shown anywhere on the overview or settings page for a project.

issa · September 4, 2019, 9:55pm

okay, noted again! i will add an item for the future to add a briefcase access panel/link/etc of some kind.

i'll also see if i can add explicit links for those emails. right now it's just up to the email client to figure out the link (small team, limited resources, etc).

dr_michaelmarks · September 4, 2019, 10:09pm

No problem @issa
Looking forward to discussing it I hope at the forthcoming convening.
My aim is really just to test Central to death because we use ODK so much at LSHTM and we want to start moving to this as our server base.

Question
Any way to download the server audit log? (This is vaguely related to other stuff I'm working on with @LN and others about tracking every change in a project in relation to research standards etc)

And/or filter the audit log by a specific project - or access an equivalent of it when within a specific project?

issa · September 4, 2019, 10:29pm

no server audit log download yet but i've been assuming somebody would want it eventually.

filter by project etc: the plan is eventually to introduce an audit log tab to each project / form / user which would automatically show just the log events for those entities. no idea when that plan would be executed just yet.

dr_michaelmarks · September 5, 2019, 9:37am

@issa @yanokwa
Discussing today with @chrissyhroberts about granularity of user controls
Currently the Administrator is effectively a super-user (can do everything)

Coming as I understand in 0.7 is a Viewer only role (I presume at the Project level) which is great and also more granularity at the app user level - also great.

We were discussing if there should be a lower-level administrator role
i.e

I can create new projects
I can create new non-admin web-users and assign them as the project manager
I do NOT have the access rights to edit forms, view project that the current super-user admin has

The advantage of this is it allows a low level admin to create projects and hand them over to individual researchers without the low-level admin being able to play about with someones project

Looking at the abilities laid out on:
https://docs.opendatakit.org/central-users/

We would be saying

Projects
Create: YES
Edit Details: YES
Archive: OPTIONAL

PROJECT FORMS:
No to all options

PROJECT FORM SUBMISSIONS:
No to all options

PROJECT APP USERS:
No to all options

WEB USERS:
Probably yes to all options (although maybe not able to make someone an admin)

Server Config:
Probably no to all options

Florian_May · September 5, 2019, 10:39am

If R is an option, you can access audit logs via
https://dbca-wa.github.io/ruODK/reference/audit_get.html

Next you might want to filter actions by those involving projects, then parse them into something readable (rectangular).

What is a question that project audit logs could answer?

MatthewMac · September 5, 2019, 10:40am

@dr_michaelmarks I've now enabled the server to send external mail through our internal relay.

dr_michaelmarks · September 5, 2019, 4:20pm

In GCP compliance (for medical research) its useful to be able to track things that happen to the database so project level audit logs are helpful for that.

issa · September 5, 2019, 6:07pm

The advantage of this is it allows a low level admin to create projects and hand them over to individual researchers without the low-level admin being able to play about with someones project

But what's to stop this person from creating a dummy user on another email account, assigning it manager on the project anyway, and doing with the project what they will? And if they wouldn't because honor system, then why not make them a full administrator?

In general, Central follows a paradigm many (most? all?) permissions systems follow, which is that no user is ever allowed to grant a right that the user themself does not possess.

Florian_May · September 6, 2019, 2:51am

E.g. listing updates to projects (details in column "details"):

Or get entire log and filter later:

dr_michaelmarks · September 6, 2019, 2:50pm

Now that I have the Audit functionality working (having fixed a quirk of XLS > XML conversion if using old version XLSForm Offline)

Now trying to get Audit download working well from Central 0.6

When I use Download all records button the Audit.csv file doesn't seem to have all the audit trails for each individual instance. I only seem to get an audit.csv file containing the initial submission audit trail.

When I download from central via Briefcase these do all get concatenated into a single CSV plus each individual CSV ends up in the media folder.

Thoughts?

LN · September 6, 2019, 3:46pm

I can't find an issue or notice so @issa will need to refresh my memory on where this is documented but it is a known issue that there is no collision resolution on export when multiple attachment files have the same name. This Briefcase issue highlights some of the challenges with collision resolution.

Given that Central 0.7 will produce the same single big CSV export as Briefcase (see the release criteria), do you feel like you still would like to export the individual audit files? If so, do you have a sense of how you'd like them to be named?

issa · September 6, 2019, 5:39pm

i haven't documented this particular quirk anywhere, mostly because i had no idea it existed but also because it sounds like it is not a Central-related issue. either way, at least 0.7 should resolve this one.

dr_michaelmarks · September 6, 2019, 7:33pm

Because central let's you download the individual audit trail meta data if needed I'd be happy if the download everything button gave you one CSV with all changes together.