Approaches for linking form instance changes to individuals

Correct, although its not quite so bad to continuously logs value changes when moving a slider. But you certainly dont want to pop up and block on a 'Reason for change?' window every time... [I'm trying to think how on earth I'd implement this tracking for my current sliders :anguished: ]

I violently agree :grin:

1 Like

I agree with the fact that many minor changes don't need a reason. As I said in the call, ideally the reason for change should be requested at question level. It would be great if in the xls form we could specifify for a question if a reason for change is required or not.
Depending on the CT (let's say more or less strict) it is not needed a reason for all changes.
But if we can have compulsory to date & identify who made the change that is OK.

I didn't react "really strongly" :slight_smile: I just just said that if it is a choice this should not be listed as a "technical limitation" .

@dr_michaelmarks Are you going to the ECTMIH in Liverpool? I'll be there and present the ODK implementation of the WHO VA. It would be cool if you are around and we could meet in person.

1 Like

Yes I'll be there on the Thursday and Friday

2 Likes

Thanks for inviting me to the TSC call and to all TSC members for being so thoughtful in your feedback.

On the call, we decided the next step would be for @aurdipas to get one last round of feedback from his team. I also mentioned I'd show a sample interaction possibly from a form @aurdipas provided.

I went ahead and built a quick example with a simple toy form. Please note that this shows the interaction in a one-question-per-screen context. There will need to be an addition made to the spec for clients that show multiple questions per screen by default. See conversation at Define audit log event behavior when there are multiple questions on one screen.

This is an extremely rough prototype so the dialogs are unstyled, the text is not polished, the question shouldn't change until the reason for change dialog is dismissed, etc. What it does show is the flow intended by the current spec. You can also see the resulting audit log with a description of how events relate to the video in this Google Sheet which also contains the blank form.

Its probably worth making explicit (since I'm not sure it has yet?) that the sticking point seems to be around specifically when to prompt the user to enter change reasons.

Requiring the enumerator to enter their id/name at the start (and assume an implied opt-out if they dont?), and then logging this info and a timestamp whenever anything changes (including blank-to-non-blank!), is probably a no-brainer; its a minimal amount of additional audit data, and it happens entirely behind-the-scenes on the client. However, what is important is when we will interrupt the user while they are in the midst of filling in a form, and require them to enter a RFC (Reason For Change), eg via a popup.

I'm not saying this gets us any closer to answering the question (alas) but maybe it'll help focus on the core issue? IMO its not quite so much "When to link form instance changes to individuals", as it is "When MUST we interrupt users to enter RFCs".

there's no link to a google sheet :frowning:

Yikes! Fixed. :see_no_evil:

Yes, thank you for making that really explicit. With that framing, the little demo video I put together and the resulting audit log is meant to show what it looks like when the reason for change is requested when any non-blank value is changed.

Another good point. I believe this means we can file issues and start work on the odk:identify-user portion of the spec since it is now entirely disaggregated from reasons for change. Does that sound right?

Yes, I believe we can probably disaggregate logging user + timestamp from RFC's in the change log, and make these low hanging fruit (so we can at least make some progress whilst struggling with RFC's...). But whether we should I'd probably still want to hear back from some users of this feature if they see any problem with this (@dr_michaelmarks? @aurdipas?). The only potential issue I can foresee is that you could well end up with entries in your change log without an associated reason; although it should be a simple matter to filter these out.

Do note that timestamp and change values are already in the spec and supported by Collect as described in https://opendatakit.github.io/xforms-spec/#audit-attributes and https://docs.opendatakit.org/form-audit-log/#change-tracking.

The spec additions for identity tracking would be an odk:identify-user audit attribute and a user column in the log. Clients that support odk:identify-user would be responsible for getting a string user identifier and including that identifier with each event. I believe that in prior discussions we agreed that this would be useful even without reasons for change (I feel like maybe you even suggested the disaggregation initially!).

1 Like

I have been digging in to this a bit more.

Here is an example of the change log that redcap (widely considered GCP compliant) makes

As you will see if captures username, time date and change.

As far as I can see there is actually NO function in redcap via the online or mobile interface to capture reasons for change.

The following is taken from a redcap FAQ page:
"For each event that changes data in the database, REDCap records the time and date, the username of the person logged in at the time, the type of event, and the changes made. The entire audit trail is stored during the lifecycle of the database."

So it seems to me that any Reason for Change data we collect will be beyond what most people do.

I would therefore suggest we record it at the level of the whole form. I.e not per question or per screen.

@LN can you remind me what triggers an audit log capture? Definitely if you move off the screen but can you remind me if it captures changes made whilst still on the screen (i.e i type 7, delete and type 6) (I'd think that was overkill).

I got a feedback from our Datamanager in the Med department.

"Regarding the spec, all fine except as I commented before:

User not having to login with a password to confirm identity is a major weakness and for this reason the ODK audit trail/reason for change is not suitable for 'electronic' clinical trials. "

But this we knew as a limitation. ODK will be let's say "paper equivalent". But if we consider that new regulation for Clinical Trial are now moving to Electronic data capturing tools, if at one point ODK collect will allow user login with password this will be super :smiley:

On the other hand I got the attached doc.

User Story_Reason_for_change.docx (12.9 KB)

Have a great weekend
Aurelio

@dr_michaelmarks see you soon in Liverpool

Technically this is not required by GCP.

GCP states

5.5.3 When using electronic trial data handling and/or remote electronic trial data systems, the sponsor should:

(a) Ensure and document that the electronic data processing system(s) conforms to the sponsor’s established requirements for completeness, accuracy, reliability, and consistent intended performance (i.e. validation).

(b) Maintains SOPs for using these systems.

(c) Ensure that the systems are designed to permit data changes in such a way that the data changes are documented and that there is no deletion of entered data (i.e. maintain an audit trail, data trail, edit trail).

(d) Maintain a security system that prevents unauthorized access to the data.
(e) Maintain a list of the individuals who are authorized to make data changes (see 4.1.5 and 4.9.3).

(f) Maintain adequate backup of the data.

(g) Safeguard the blinding, if any (e.g. maintain the blinding during data entry and processing).

In fact the word password occurs at no point ever in the GCP regulations.

My question @aurdipas for your data manager would be

"Where is the password for paper"

We propose to get round this with other approaches

  1. Device level locking
  2. If necessary one username/password per device
    i.e
    Different Collect account for every fieldworker

this means to me that a sort of password protected. But I'll ask our data manager to be more precise.

FDA is also expecting that

Access must be limited to authorized
individuals
• Each user should have an individual
account/password
• Passwords should be changed at
established intervals
• The system should limit and record the
number of unauthorized log-in attempts
• Automatic log off for long idle periods

we are not on paper we use an electronic tool for data collection Macro

And our data manager agreed that this system is paper equivalent (without password).

I'll ask more details to him as soon as possible and share with you.

Yes so we propose that you can achieve:

(d) Maintain a security system that prevents unauthorized access to the data.

At the device level - just password protect the phone/tablet.

Maintain a list of the individuals who are authorized to make data changes
This is normally at the server level I think; i.e who can change the raw data.
But could also be done by giving each data collector a unique username.

the login/password is a wish for me in the future.

For the moment being paper equivalent is OK.

this means a tablet per fieldworker that can not be used from someone else or you have multiple users per tablet?

The limitation of not having a login is also that the user that first fill the form is not tracked (the reason for change it happens for a modification not for first fill the form).

not necessarily at server level. We are talking also of access to the tablet.
And giving each data collector a username does not suppose a sort of login?

We can continue the discussion on Thursday next week with a beer, I pay the first one :smiley:

I think between this, and longitudinal studies, we may want to consider moving the convening to a brewpub! :grin:

This is certainly possible but it seems like it would then be worse than paper. I think from a user experience standpoint, it's also easier to explain a change right after it has been made rather than explaining multiple changes at the end. Are you suggesting this as a simplification or do you see an advantage to it?

In a one-question-per-screen context (Collect's default), changes while on the same screen are not captured. You can see this in action at second 9 of the little demo I built:

The resulting log would look like this.

I'm looking forward to seeing the notes from that conversation! :wink::beers:

I can see pros and cons of per question and per form reason for change recording. It's fidelity Vs intrusiveness i guess.

So “Why are you editing this form?” vs “Why are you changing this response?”... Former is certainly a lower-hanging fruit, one which might be accomplished via a simple additional field when prompting for ‘username’.

But would this actually be sufficient for most/some/any usecases requiring “change tracking” ?!

It is all in the eye of the beholder.
For what it is worth Redcap (considered GCP compliant) doesnt appear to record reason for change at all, only who made change, when they made it, and what change was made.