Collect will need to stop using /sdcard/odk for files

We've been making good progress on the migration implementation.

@danbjoseph thanks for your response and for the enthusiasm around getting some of the current OpenMapKit functionality into Collect.

Thanks. I've circled back with @Grzesiek2010 and @seadowg and we've agreed not to have any special behavior for new installs. Everyone will need to opt-in.

Thanks. I am working on some UI options to share soon. I'll share them in the Collect Slack first so that @Grzesiek2010 and @seadowg you can provide a first round of feedback.

I've also put some thought into designing an official API for external apps to supply form definitions and form submissions to Collect or to get XML form definitions or submissions from Collect. Some questions I am wrestling with and am interested in feedback about:

  • should we immediately replace existing access to form and submission XML? We could say e.g. we don't have clear demand for a replacement API since we don't yet have examples of actively deployed apps that are requesting this continued support. (I lean towards providing a replacement API but it's not entirely trivial and it does mean another integration point to support moving forward)
  • if we do, whether this replacement needs to be announced and perhaps released BEFORE users start having file migrations so that folks with app integrations can react
  • should we bump the version to Collect 2.0 when we start rolling out the file migration? For developers, this will make it very clear that there are breaking changes. For most users, this would seem confusing because there won't be major visible changes. Since folks generally auto-upgrade through the Play Store, the more significant version change probably wouldn't make a difference for most.

I have identified two other applications with integrations that will be affected:

  • OpenHDS @aurdipas do you know of active deployments? The only one I am aware of is the one @Batkinson works on but they have now forked both the OpenHDS mobile client and Collect. @Batkinson do you expect to stay on your forks indefinitely? Do you expect to address the scoped storage requirement or deliver outside of the Play Store once it is enforced?
  • Project Buendia I've been talking to @zestyping and he has agreed to review a new API. The project uses a Collect fork currently so would not be immediately affected.

Both are examples of applications that improve longitudinal data workflows and add communication with some custom source of entity (patient) data. I'm certain there are many other such applications out there but the big questions is whether they are in active use and would leverage a new API if one were introduced.

@TSC-1, I think this is something you will likely want to discuss during your next meeting.

Please take note @adam.butler (next facilitator)

OpenHDS is the latest development and it is used in (at least) 7 HDSS sites.

2 Likes

Ahhh, of course, I should have looked at the Swiss TPH Github. Thank you for that information. In that case I think we do need to provide an alternate API as soon as possible. Is the OpenHDS app deployed through the Play Store?

1 Like

Just to make sure it nothing slip thru cracks, we should test last-saved doesn't break (since it's retrieving previous submission from here, right @ln?)

1 Like

This Google doc has a proposed user experience. Please comment directly inline in the doc if you have feedback. I'm particularly eager to hear ideas for simplifying things for enumerators.

I have also shared that doc and a summary in a features thread at Let users know of required storage location change and let them opt-in before August 2020 in the hopes of getting feedback from folks who don't watch the development category.

Yes, that's correct. :pray:

nope. Is not on the playstore. And is not maintained/developed anymore due to lack of funding. As far as I know it works max with Android 7 and we suggested to openHDS users to don't update anymore ODK Collect to avoid issues.

2 Likes

Apologies for the delayed response. I couldn't get to this until now due to competing priorities.

TLDR: The forks aren't going away. We likely will address the scoped storage requirement, but not necessarily the same way as mainline Collect.

What meaning does the "now" in "now forked" have? The fork of openhds tablet happened circa-2015 and the fork of collect only just happened within the last few months. They are also very different in nature.

After working with original team from the University of Southern Maine, I eventually took over development of their existing fork of openhds for the same project on Bioko Island. I just "production"-ized it by giving it a home, branding it, and fixing its major flaws to function for its intended purpose (which was not the same as openhds). There wasn't an active online openhds development community to integrate back to, and many of the changes that needed to happen were not clear conceptual fits with a demographic surveillance system. To my knowledge, it is the only actively developed fork of that project, including the original.

The collect fork happened for different reasons. Mainly, instability of updates from Google Play kept breaking our daily operation of around 50-100 devices. However, I had been reluctantly considering the need prior to that, for similar reasons to the openhds fork: its purpose is not really the same as collect. Many features that would not make sense in Collect (as it exists today) make perfect sense for the fork. For that reason I don't see it going away, at least until the project does. However, we want to track as closely with mainline as possible, both to keep maintenance reasonable, as well as to continue to benefit from all of the great work the ODK community is doing.

Yes, we will likely be adressing the scoped storage requirement for CIMS Forms within the next couple of weeks. We are interested in a Collect solution, but don't expect direct support of any kind (including expedited delivery of an api) from the ODK community. This will likely be bundled with changes to way the apps integrate, since the Collect model had some undesirable quirks that we will want to address.

I can't speak to Project Buendia, but we are not solely tracking patients. We actually track households and inhabitants on Bioko Island for accurate coverage reporting and other metrics. We also have plans on expanding that support to track arbitrary entities, since different organizational units want to leverage the same shared database during inital data collection for easier analytics downstream. For example, tracking bednets, informal drug dispensaries, private clinics, etc.

Happy to expand on or clarify any of this if anyone is interested.

Thanks for the context, @Batkinson. If you're up for it, I propose we hop on a quick call to discuss the scoped storage and app integration changes we're both planning. It's likely we'll have to do at least some of the same work so I think it would be beneficial to coordinate a little bit, especially since you hope to track the Collect mainline. I'll follow up with you directly and I can report back on this thread.

In the mean time, I have also checked in with @Jason_Rogena and @Kigamba at Ona and they have confirmed that they continue to maintain app integrations through the /sdcard/odk/ folder.

I had a great conversation with @Batkinson about this and a number of topics of common interest. After Wednesday's TSC call, the plan is to wait at least until Collect v.1.27 before releasing an API so that we can learn more about the needs of real users with integrations. @Batkinson found that reasonable. In the mean time, he and I will discuss any tentative plans we may have for the replacement API and try to make sure that the solution will be reusable for all.

Thanks, @Batkinson!

The migration implementation is now in beta. Learn more and try it out at ODK Collect v1.26 Beta 1: scoped storage migration.

2 Likes

Hi @ln,

Sorry for not replying, it seems my notifications are off & I'm late to the changes to getodk :smiley:

Ona is about to start adapting some of our apps, WHO Steps app, to using scoped storage & ensuring that the integrations to ODK Collect are updated. I found some code that generates a CSV used for the pulldata function in the form. We were solving the CSV media file hackily by adding it to the ODK form media folder. It seems that this is the only feature that does not use the ContentProvider or the activity hooks.

Do you have any ideas off the top of your head? If not, I can get back with several options later & we can see what would be good.

I don't have this in my brain anymore so definitely take my thoughts here with a grain of salt.

I think the general approach would be for external apps to hand Collect URI(s) to form definition and/or media files with permissions through an intent extra. I vaguely remember giving some thought to where this should go and I think my conclusion was that it should be a new, dedicated activity that takes whatever URI(s), copies the files into the Collect directory, registers them in the database and generally gets them ready for use.

Even though you're only looking to register media files (in this case a CSV) with Collect, I think the solution design should also include registering a form definition. Those are tightly related and we'd want to make sure that they're handled coherently. Additionally, whatever we come up with, we should then later be able to add something symmetrical for handing submissions to Collect.

Would be great to see some of your ideas.

This is a much more well thought out & it makes clear sense. Let me collect my thoughts and see what I can come up with

1 Like

@Kigamba, here is the release timeline we're looking at:

  • v1.28 released ~mid August with message that v1.29 will force the migration
  • v1.29 released ~mid October that forces the migration on first app launch

You can read more in the Collect issue to change the banner text.

That means if you do still want to add support for external applications handing form definitions and media files to Collect, there should be a proposed approach by mid-August at the latest (in time for v1.29 planning). A PR in mid-September would be ideal so there's plenty of time to go through quality assurance. The last chance to get a PR in would probably be the last week of September.

2 Likes

is that why I have been unable to submit my data since yesterday? (9/07/2020)

@LN Thanks for the update. I'm working on getting resources allocated so that I can start on this. I think that the September deadline is considerate enough. The August timeline for proposing an approach also sounds good. Sorry for the silence on my end. I'll communicate in due time. Thanks again for the update

1 Like

Hi @Agumash
I can assure you it's not related. Maybe there is something wrong with your server. Please create a separate topic if you can't solve your problem by yourself.

BTW welcome to the forum! Please introduce yourself here!

Welcome to the community @Agumash

@LN :disappointed: I couldn't get the resources to work on it now. We'll reach out when the situation changes & I get a go-ahead

:smiley: We also discovered that the files are accessible for some android versions inside the /data folder. However, they are private on Android 9. The support team & some users tend to copy over the failed submissions for sending with ODK Briefcast on a computer