Send submissions via SMS

It looks good to me so far. Lots of clear use cases and guides here. I have a few questions though.

  1. Will this spec govern this current implementation where the text being sent will just be shortened and be in a human readable form vs it being compressed and received by a server for processing? Just checking to see if there should be any design considerations to support this.

  2. How will complex types be handled? Such as image submissions. Will the form client be responsible for enforcing constraints on these types? Or will the be enforced from the form design phase so certain types can't be tagged. Or is it a situation where even though an image can't be submitted the answer could still tell if an image is present or not.

  3. How will we go about representing normal groups and repeat groups. WIll we have a unique identifier for them eg. G or RG before the info is should in the SMS?

  4. What are the rules in terms of truncation? Should form answers be truncated or should an answer only be included in a SMS if it can fit within the defined length? Also should the form identifier contain any means of informing which part of the form this is? such as something appended to the form ID so that even if truncation takes place an user would be able to match the submissions easily?

Ha, I'm glad you are appreciating the feedback! :smile:

Your suggestion to select the transport method and have the option for both seems perfect!

1 Like

Here's what I think (but not necessarily what is correct)! @alxndrsn used to work on FrontlineSMS so maybe he has some ideas.

  1. Let's focus on the text being sent for now, but I do think we'll eventually add compressed SMSes, but my gut is that won't require huge changes in Collect. Is there a particular issue that you are worried about?

  2. This is probably a little user-hostile, but I think that Collect should send any non-group question that has a tag, even if it has a binary type. I argue that this is reasonable because the XML submission that Collect generates is text (in the case of images, it's the name of the file). We can manage the hostility by adding warnings to the form designers.

  3. Groups and repeats. This one is tricky. Since I'm proposing we only allow tags on non-group questions, then we don't have to worry about groups themselves, just the questions inside the groups. For repeats, an ordered list of the elements should do it.

    <children>
     <child tag="CH">robert</child>
     	<colors>
     		<color tag="CO">red</color>
     		<color tag="CO">green</color>
     	</colors>
     <child tag="CH">sally</child>
    </children>
    <state tag="ST">CA</state>
    

    Then the SMS would look like this +CH robert +CO red +CO green +CH sally +ST CA

  4. No truncation of answers. Whatever gets tagged gets sent. Form designers already have a way to control the length of text by adding constraints and selects by shortening the value that is persisted. As to form parts, that's a great point! Every message should have a "header" with form ID/prefix, an instance ID, and the sequence number

I'm sure I haven't thought of most of the gotchas, so please keep the questions and concerns coming!

2 Likes

Another requirement (apologies if that was implied already: Resending any failed messages.

Why? In many places where this is going to be used network failures to send texts are extremely common, especially in areas with spotty connectivity, bad weather, network outages, etc. Unless Collect has a way to check if a message was successfully sent via the network and resend all failed messages (forever, until it's finally sent), we could end up losing a lot of data.

2 Likes

Personally I'd be against embeddeing sequence number in the messages as multipart SMS should handle these transparently.

For instance ID, does this refer to a unique ID for each message? If so, I'd prefer if this were optional: for some datasets identical form submissions can safely be ignored, and an ID would reduce space available for the form data itself.

An optional CDMA compatability mode (ref https://github.com/opendatakit/collect/issues/242#issuecomment-382119352) might be a useful extension to this work - this would cut each SMS down to 140 characters, and handle multipart splitting and inclusion of a sequence header

Spec looks awesome so far. I just had one question though, are we thinking about escaping the value of the delimiter and the shortname if they're presented as values for the questions and what implication that will have on the number of characters in an SMS?

@Jason_Rogena Glad to see you joining the fray!

Assuming a space delimiter, I don't think you need to escape it. You just read everything between +TAGX and +TAGY and consider that the answer. Does that sound correct to you?

I think you are correct about escaping the shortname though.

  • If a person's name was "John Doe", the submission would be: F123 +N JOHN DOE +G M +A 35.
  • If the name of the person was "John +D Oe", the submission would be: F123 +N JOHN +D OE +G M +A 35 and that'd be a mess.

Do you have an idea of how we solve this problem? Maybe we prepend a backslash (\) so it's F123 +N JOHN \+D OE +G M +A 35?

@alxndrsn If you are confident that multipart would handle the multiple parts seamlessly across most providers in Sub-Saharan Africa, then we can drop the sequence number. Can you confirm this?

The instanceID that will be going in the SMS refers to the unique ID for each submission. If what you say above about multipart is true, then it only needs to be in one message. I do think it's needed in at least one message or you can't tell two submissions with the same value (e.g., two patients named Bob) apart.

As to CDMA compatibility mode, https://developer.android.com/reference/android/telephony/package-summary.html suggests that Android handles GSM and CDMA transparently.

@Tino_Kreutzer Android can detect if a message has been sent and so Collect will know.

I do not think we should resend though. Mostly because I have yet to see the iPhone or Android do this on their SMS apps. My guess is that sending generally fails when there is poor network and when the network is poor, constant retries will eat battery.

Also, the Collect send screen is blocking, so enumerators generally have to hit send and wait for the dialog to confirm. So they'll know what failures are occuring.

What do others thing of auto-resending?

Collect's use case is different from a normal SMS app since it's using sms for submissions so auto resending would make it resilient once implemented properly. With poor network connections, I thought of a way to detect when a device has poor network connectivity by observing the signal strength and resending once the connectivity is good again. There would be a window for this though to ensure it's not battery intensive. This feature would come in handy if a user has many sumissions that are going via SMS. They could focus on collecting data knowing the app will retry when the conditions are right. If something like this isn't necessary it could just work the normal way where once a message fails due to poor network connectivity the user has to manually press send again to retry.

@yanokwa I agree with you, as long as the submission remains on the to-be-sent screen even if only one of the multi-part messages didn't get sent (in this case it's no different than if a WiFi upload failed halfway through).

There is a case though for background sending that's different from the current UI, especially because it could take so much longer to transmit hundreds of SMS.

1 Like

@yanokwa Thanks. Both your suggestions sound reasonable in my opinion. I guess compression should deal with the escape character's data overhead pretty well (techniques based on Huffman Coding handle repeated characters pretty well) if it ever becomes a problem.

@yanokwa This is all looking great so far. I like the suggestion of having the dropdown menu to choose the transport!

Hey everyone!
So I have been working on the feature and it's coming along nicely so far. I have an idea of the changes that could be made to the form uploading screen and I wanted to get some feedback on the design.

So I just tweaked the original UI a bit so that both submission types would play well with each other.

  • Success and failure bars are shown beneath each form while the submission is taking place along with a status message about the submission.
  • You are able to cancel an individual submission i.e if messages haven't been sent you can stop them.
    Along with that, I was thinking if a submission is stopped then the x button would change back to the checkbox so that you can select it again and press the Send button below similarly to how you would upload a normal form.
    Reviews welcome :slight_smile:
2 Likes

That's looking really slick, @joeldean. The "Collect way" would be to show status in a dialog and block the UI. I think part of the reason that approach has stuck even though it's not modern or slick is that it ensures enumerators have to see the result of their send. In your UI, what happens if a form fails? Would there be a dialog or notification guiding the enumerator back to take action?

@joeldean this is awesome! Great work

2 Likes

Yes, if they aren't on this screen a notification would be shown that would guide them back to take action.I was thinking that an enumerator would be able to start submissions and continue what they are doing elsewhere eg. completing other forms and be notified when it's completed instead of waiting on that screen.
Also I have seen more failures when submitting sms than when making a direct upload once I have a good internet connection so allowing the submission progress to take place in this manner allows the failures to be done a bit more elegantly.

1 Like

I like it and I think it's a model we can move towards in other places where there are currently blocking actions.

It would be good to get some confirmation from users like @barhin that this change would not cause too much confusion for data collectors.

Paging @Shobhit_Agarwal who had made some demos in that direction last summer and thinks a lot about UI in general and @tomsmyth who also thinks a lot about user experience and making sure the app is intuitive.

This is a really clever implementation, @joeldean! I love it. My only minor complaint is with the color choice and size of indicator. Red/green is a natural choice, but about 5% of the world's population is color blind, so I'd just go with a single color (Android blue) and stop it half way when it has failed.

No problem! Changes made. Good to see that you have an eye for accessibility :smiley: