Proposal: specify a fixed image to be annotated or signed

Sign or annotate a fixed image specified in the form definition describes a feature that would allow form designers to specify an image to be annotated or signed.

There have been two different approaches proposed for adding support for this in the ODK XForms spec.

Use defaults

One idea brought up by @Grzesiek2010 in https://github.com/opendatakit/collect/pull/2428 is to use the standard default value mechanism. This doesn't currently work for binary files (as described below) but it could be a way to support annotation/signature of fixed images without adding any form spec changes.

As stated in the specification, "Any value inside a primary instance is considered a default value for that question. If that node has a corresponding input element that value will be displayed to the user when the question is rendered." However, this is not currently true for questions of type binary. This is because clients typically (always?) see the following as different and saves them in different places:

  • media attached to the form that can be used to, for example, illustrate a question or lookup a value
  • media attached to a filled form that will be sent to the server with a submission

That means that if a form's primary instance contains the following: <my_image>my_face.jpg</my_image>, the client won't find my_face.jpg in media attached to the current filled form and so the default value won't be recognized. This is true even if the user uploads a media file with filename my_face.jpg alongside the form definition. However, if the user selects an image called my_face.jpg from their client, <my_image>my_face.jpg</my_image> will now be a valid part of the primary instance.

It's weird that defaults work for all types except for media (binary). One way to make this more consistent would be for clients, when they see a question of binary type with a filename value, to look in the directory for the current filled form for that image (current behavior) and if it's not found, look in the directory for media attached to the corresponding blank form and make a copy to the filled form directory.

One question about this is whether defaults should also be allowed for audio, video, raw images, etc. I think so for consistency even though it's unlikely to be all that useful.

Use the image label or introduce a new attribute

As described by Mitch in https://github.com/opendatakit/collect/issues/84. This would more closely match how media files are added to a form for illustration purposes (images, audio, video).

Another related option would be to introduce an appearance such as markup that combines with annotate or signature and hides the buttons to select or capture a new image while also using the label image for the annotation.

No matter the spec option we choose, we'll probably want an appearance to hide the images for taking and choosing images.

1 Like

Thanks for this @ln! It is very helpful to me to see this written out.

I think the default-value-with-image-file approach is possible but to avoid magic to pass image files with an empty form, we would need to introduce a new datatype (e.g. anyURI or base64Binary). For the former, I think we could use the jr:// scheme to retrieve the file from the manifest and submit as base64Binary. The latter would be a little easier to implement on the client. Orbeon supports [something like that]((some example here).

The challenge is probably to make this easy to use for users when designing and uploading a form and analyzing the submitted data (edited images). It would probably require some fancy processing in pyxform and especially in Aggregate/Central/Ona/KoBoToolbox.

Otherwise, to keep things simple for pyxform, Aggregate, Ona, KoBo, I'm in favor of a more hacky but not terrible approach using a new appearance and question label (tbd exactly how).

Always good to dive into these tricky spec issues with you, @martijnr!

To give a concrete example, the form definition would have a node that looks like <my_image>jr://images/my_face.jpg</my_image>, the question widget would copy that file, allow the user to modify it, and then save <my_image>my_face.jpg</my_image> (or perhaps <my_image>new_filename_generated_by_widget.jpg</my_image>)? I don't find this terribly offensive but I'm still not convinced it's better than what I was describing.

The base64Binary option seems like a convenience option not to have to separate the form and its media. I think we can consider that separately.

Agreed this is generally an important thing to aim for. That said, if a blank form's instance has <my_image>my_face.jpg</my_image> and its corresponding manifest contains a my_face.jpg file, that doesn't seem like magic to me. I find it rather consistent, again given a slight change in client rules in where media described in an instance is fetched from. That is, the client would look first in the instance media storage location with a fallback to the form media storage location.

I see that you want to maintain the conceptual distinction between media attached with the blank form and media attached with a filled form but it's not clear to me why it's such a critical distinction. It doesn't seem consistent with the way other defaults work and it seems that with the approach you've described, a binary node with a default would look different from a binary node with a value set by the client. This feels odd (though not terrible).

What am I missing?!

@issa, it would be great to get your thoughts on this since you recently worked on parsing ODK XForms to extract a list of required media files for Central's media attachment feature. Do you have a sense of what would seem most consistent and least magical to you?

My impression is that the default values approach feels very aesthetically nice from a math perspective, but I'm not honestly sure what problem it solves besides enabling this case. What else would default binary values be used for? Why would we do the work of downloading all that data just to replace it?

I'm also not very clear on the suggested datatype, or how the filesystem on this thing actual works—is it shared between form-world and submission-world? Or would the default value be special-cased to look at form media and applied values look to submission media?

Are there other combinatorics to worry about by introducing default values for binary fields? I don't know a lot about XForms but I know it's ridiculously interconnected.

From the perspective you were after, either way it's a new detection case. As much as it seems like the messy all-of-the-things-all-over option, it's truly not offensive from a detection perspective to just hunt down some particular binding or input attribute for a filename.

1 Like

Agreed.

I dont quite see why this cant be accomplished by simply 'prefilling' the form with defaults - in this case an image - and having some (appearance?) flag to indicate that instead of replacing the (image) 'value', instead you want to mark-it-up/edit the existing 'value'. Indeed, you probably want to support basically the same usecase anyway, to allow users to markup a photo after they've taken it [although I agree that marking up a previously captured signature doesn't make a lof of sense... But in this case the previous 'signature' would actually be a background picture].

1 Like

I agree that would be odd. With the new datatype I was suggesting to store the new value in a new way as well. The new way would be as a URI or base64Binary. I think anyURI, would cover both jr://image/path/to/image.jpg as well as base64 (but am not sure).

Could you please share an example of what the XML would look like? I need something concrete to sink my teeth into! :smile:

A user would have to explicitly decide to set a default value and attach an image so I don't think there would ever be a case where downloading the data would be a "waste." Are you thinking of a scenario?

I'm imagining a change in client lookup rules when a node with a binary type binding in the instance has a value. Right now clients only look in the submission world. The change I propose is for them to still look in the submission world first but if they don't find something with that filename there, they look in the form world and copy the asset to the submission world. This maps to my mental model of what a default is.

I can't think of anything but perhaps you can give me an example of something like that you've seen before?

Can you say more about why that flag would be necessary? The way I see it, we'd just exactly match what happens now if you open an existing filled form and it has a binary file associated with it. That is, the question would be populated with the contents of that file and the user could either continue marking that up or replace it with a new "canvas" to mark up.

For a signature, if there's no background image then you when you 'edit' the signature you basically want to replace the image with a new signature image (not markup the previous signature, which would be strange). But if there is a background image then you do want to markup the image (with your signature). Without a flag or something I dont quite see how you can determine whether any existing image is a background (to markup) or a previous signature (to replace)?

Alternatively, every time you open up an existing image (signature or otherwise) you have to prompt whether the replace or markup, before proceeding, which sounds like what's happening now.

Currently signature and markup have the same behavior: when you edit an existing value, you modify what's already there. This is consistent with how paper or other kinds of digital signature interfaces work -- if you're like me and often lose the bar on the t in your last name in your signature, you can always go back and add it in. You can also erase what's there and start over.

Seems a bit strange to me to 'markup' a signature, but for simplicity (and consistency) it probably no biggie. As you said, you can always hit "Clear" to explicitly replace anything already there

1 Like