XLSForm spec proposal: add syntax to make it easy to use a value from the last saved instance

My experience is that non-developers can spend a LONG time hunting for the pipe character on their keyboards and/or use a capital i or a lowercase L and get very confused. It's especially confusing because some keyboards have it labeled as a split pipe (¦).

An additional requirement: whatever characters need to be typed should be recognizable to anyone.

That's a good point about the split pipe, @LN - I also think that # is a good option. I've been thinking about @Tino_Kreutzer's idea of thinking forward to possible case management use cases. It seems like it would be helpful to come up with a generic way of saying "the value of field x of entity y". In this particular example, x = "street" and y = "the last saved form". In a possible case management scenario, we would probably want to reference fields on the root entity (where "root entity" means e.g. "the patient I'm reporting on", or "the tree that I return to measure every month") (NB I'm not saying that "root entity" is the best name for this, just using it for the sake of these examples!).

In the case management scenario, we would probably want to reference these fields in labels (e.g. "what is 's temperature?") or skip logic (e.g. "skip the next question if the tree is partially in shade"), rather than default values, so that's worth bearing in mind: it would be good to come up with a syntax that is also usable in both of those contexts.

It seems like we all agree on ${...} as being a reasonable way of representing "the value of", so now we just need to work out the best way to say "field x of entity y". The two options we've talked about are x#y and y(x), but I wonder whether it might also be worth considering something along the lines of y#x, as @LN suggested?

Possible values of y would be pre-defined, such as last-saved or root-entity (or more succinctly, last and entity). So in the examples that we have:

  1. Last filled street: ${street#last} or ${last#street} (it might be worth allowing an abbreviation if the field name being accessed is the same as the current field name, so then it would be ${#last} or ${last#}
  2. Patient name in label: what is ${full_name#entity}'s temperature? or what is ${entity#full_name}'s temperature?
  3. Tree status in skip logic: ${partial_shade#entity}=yes or ${entity#partial_shade}=yes

I think my vote would go to ${y#x}, i.e. ${last#street} / ${entity#full_name} / ${entity#partial_shade}

1 Like

@adam.butler I like where you're going.

Here's what I'm understanding for ${entity#full_name}:

  • there'd be some kind of standard identifier (e.g. recordId) linking the current form instance to info previously collected about the entity this form instance is concerned with. Presumably, the previously-collected info would be in an external secondary instance representing all entities and their info.
  • the XForm would define a __entities instance to give access to all of the entities' info
  • entity in ${entity#full_name} would expand to something like instance('__entities')/recordId/data
  • the entity# shortcut would only allow referring to values related to the entity this form is about, not other entities. For example, if I'm defining a form that will collect information about houses, I can use it to refer to information previously collected about a specific house but I can't use it to refer to information about the neighbor's house or an occupant of the house that info is being collected about.

Did I get that right?

I think it's a useful concept even with the limitations in my last bullet above. I'm on board for introducing ${last#<unqualified field name>} for now with the goal of expanding to other prefixed keywords like ${entity#<unqualified field name>}.

Yes, yes, yes and yes :slight_smile: - and thank you for making all of that explicit @LN !

Just as an addendum to your last bullet: if the occupants were somehow marked as being a direct attribute of the house entity, then I think that it ought to be possible to refer to them using this syntax. But that's probably a discussion for another day...

1 Like

A possible issue I see with this definition of #entity is that it appears (correct me if I’m wrong...) to tie entities to a specific instantiation of a specific form id+version (?). Whereas in the general case - and certainly in mine, you can perform multiple (and completely different) ‘inspections’ (aka fill in completely different forms) about the same ‘entity’, so this vague “entity” thingy exists independent of any particular form, yet alone specific form version.

In this context, What is an ‘entity’? Or should it instead be called say “specific form instance”?

“Entity” to me conveys a physically unique object. Whereas a form instance/submission is rather more an partial snapshot in time, unique only unto itself.

@Xiphware the way I see case management working in ODK is that you would have two different types of form: entity forms and report forms (this nomenclature is not yet written in stone). There's more information about the proposed approach here:

It's due for another round of TSC discussion in the near future.

Thanks, that makes more sense. This is actually pretty much exactly what I do in GoMobile, except we call these 'entity' forms the "Asset Form" associated with an inspection. This basically holds/displays/captures the (mostly) static data about the thing you are inspecting; eg building address, owner details, property id, etc. This model is working quite well for us in the field - every scheduled inspection also downloads the associated asset form, so all the fixed asset data (typically from the council's backend database) is available to the inspector when out in the field, and possible offline.

Indeed, one of the features I'll be implementing is adding the ability to pull in arbitrary asset data into an inspection form (for whatever reason), so I'm acutely interested in how this might be (naturally?) accomplished by being able to reference the asset data associated with an inspection via a suitably identified external instance/XPath query.

1 Like

EDITED to use last-saved instead of last as the prefix.

How have the latest ideas aged through various showers/sleeps/brain marination sessions?

Here is what I think the latest proposal looks like:

  • If the contents of the default column are not a literal value, the expression is passed through to the value attribute of the setvalue action (see documentation) triggered on first load (using the odk-instance-first-load event). Literal values get treated the same way they do now.
  • Introduce new syntax to refer to a question in the last instance: ${last-saved#<question_name>}
    • If an XLSForm definition uses this construct, <instance id="__last-saved" src="jr://instance/last-saved" /> will be added to the XForm
    • ${last-saved#<question_name>} will expand to instance('__last-saved')/data/question_name (with additional group/repeat levels as needed -- same as with the ${<question_name>} construct).
survey type name label default
text street Street ${last-saved#street}
date disaster_date Disaster date today()
integer patient_count How many patients have you seen today? if(${last-saved#patient_count} == '', 0, ${last-saved#patient_count} + 1)
select_one yes_no same_street Are you still on ${last-saved#street}?

This:

2 Likes

Yes, this looks good to me. Since we're introducing some major new syntax here, I'd like to make double triple sure that (a) this is consistent with any other similar syntaxes that already exist, and (b) it's flexible enough to cover a reasonable number of similar future requirements.

As far as I can tell, the answer to both questions is "yes", so :+1: from me.

2 Likes

Agreed, @adam.butler. Would the TSC perhaps be willing to put this on the next agenda to discuss for 5 minutes and make sure we have considered every angle and that there is consensus to proceed? This has felt like a very good discussion with most TSC members involved but you're right that since it could be potentially far-reaching, we should make sure everyone has had time to consider it carefully.

2 Likes

Wow, great discussion! I want to bring up one potential issue that I think hasn't been mentioned before, that may impact the design (if we cannot solve it in the pyxform implementation).

On the XForm side, there is a subtle difference between using <setvalue> with the odk-instance-first-load event and the current defaults (default values in instance) wrt to repeats. If a new repeat is created the current default (value in instance) would populate the default value. With dynamic defaults it wouldn't if the odk-instance-first-load event would be used (because it happens after form loading). I'm thinking introducing a new repeat-creation event with pyxform coding magic may resolve that (CommCare has jr-insert).

2 Likes

Overall, this looks great to me. Two nits to pick.

  1. The use of two underscores to prevent naming collisions is fine (seems to work for Python), but using it implicitly sets a standard for how pyxform generates these IDs. Are we sure this is the best syntax?

  2. I wonder if last is verbose enough. I can imagine a form designer thinking that it's the data from the last visit rather than the last saved. Why not just call it last-saved.

Thank you, @martijnr! That's an interesting case I certainly had not considered. Agreed it should be supported and adding a repeat-creation event seems like the best way to do it. There are already various cases in pyxform where the output is different based on nesting in a repeat so I don't think it would be too terribly big of an addition.

We should probably have a new thread for introducing and naming the event, right? JavaRosa/Collect does also support jr-insert even though it's not documented. I'm not sure how it differs from xforms-insert.

Good point. This refers to __last-saved in <instance id="__last-saved" src="jr://instance/last-saved" />. I don't think there is a case currently where pyxform has to generate an identifier. Can you think of any, @martijnr or @Ukang_a_Dickson? If there is, we should match that convention. Either way, that convention should be documented so it can be reused as needed.

I'd be happy with that.

1 Like

Can't contribute to the technical conversation but I work for MSF (Medecins Sans Frontieres) and we have implemented an ODK-based hand hygiene compliance observation tool across multiple health facilities in 15+ countries. This feature was on our priority list for future versions so observers didnt't have to keep filling in their name, location, ward, etc (in fact, discussed with @zestyping who pointed me at this thread).

So, thanks from us for taking it forward!

3 Likes

Hi, Pete! Great to see you here. This is the right place to be—it'll all get worked out on this thread; I'm guessing there will be more discussion in the coming weeks.

1 Like

@pedrito1414 if you have feedback on the XLSForm structure as shown in the sample spreadsheet in this post from a user perspective, that would be very helpful! Is it intuitive? Does it serve the needs you have in mind?

I have written up a proposal for an event to be fired when a repeat instance is created at XForms spec proposal: add event fired when new repeat instance is created. I do think it would solve the repeat issue @martijnr described here.

1 Like

I (also) dont see a compelling reason not to reuse the existing 'default' column; the contents - "foo", 1, ${q1}, today(), ... should be sufficient for pyxform to distinguish between a static vs dynamic default, and the user doesn't need to care.

1 Like

@LN The TSC has approved this approach so please proceed with the necessary issues and PRs!

Hello folks! I am not technical guy and a little bit lost while reading your conversation. Can you please, help me with following issue:

  1. In KoboToolbox prepared a logbook for vehicles. I just want to my last data entered in "arrival odometer" (after submission of form) would appear in my departure odometer of new form while filling...
    Please, find attached xls here Vehicle_Logbook_test - Copy.xlsx (17.3 KB)

dynamic_default default_to_last
${last-saved#odometer_arrival} yes

Many thanks!