Collect geotrace and geoshape improvements

ODK Collect is broadly used to conduct mapping exercises. There's a lot of potential for user experience and code quality changes for the existing geo widgets that would make them even more useful.

This isn't a single feature but I think it's important to think through proposed changes together. This will make sure the improvements are coherent and broadly useful. I have started to collect proposed changes in this Google doc and shared the document with a few users already. In particular, @ivangayton and colleagues have already implemented some of these changes in their fork. Commenting is on for everyone and if you'd like edit privileges, please message me with your email address. You can also make comments or suggestions by responding to this thread. Please share which of these changes are priorities for you and highlight any that you disagree with. Of course, please also share any items on your ODK geo wishlist!

Currently, @jamesknight is working on improving test coverage of all the widgets. This will then allow him to refactor the geo widgets to reduce redundancy and make them more consistent with each other.

One major issue that needs discussion is that the values produced by the geotrace and geoshape widgets in Collect don't match the ODK XForms specification. Specifically, geotrace is defined as "Semi-colon-separated list of at least 2 geopoints, where the last geopoint’s latitude and longitude is not equal to the first" and geoshape is defined as "Semi-colon-separated list of at least 3 geopoints, where the last geopoint’s latitude and longitude is equal to the first". Unfortunately, the geotrace widget currently produces either a line or a polygon using points based on the device's location. The geoshape widget produces a polygon using manually-entered points. I think this is an important thing to redress as soon as possible.

Paging people who have expressed interest in all things geo recently: @Aurel @David_Michallet @freddycoal @ivangayton @Jon_Nordling @mathieubossaert @SGSC @VincentD @xavier-pnm

Related GitHub issues:

Hello,

I’m glad you opened this discussion. I see posts about geowidget this summer on this great new forum (and Git). in my company we are collecting many geodatas. We deployed GeoODK few years ago with our own surveys. It’s working well. Unfortunately we had to stop to update GeoODK for some issues (unsupported search() function) in the most recents versions.
So, when ODK included the geowidgets this year we hoped it can be perfect. But the choice to develop very separated widgets for point / trace and polygone was a little surprising for us because GeoODK has a different approch.

Today i’am not a big fan of the geowidgets. I think ODK need a open geowidget which allow to the users to simply enter point / polygone / line as they want. We have many cases where we can’t specify the type of the object before the field and having 3 widgets force one more question in the survey. Not the best i think. I can add that today no widget give the opportunity to enter a line simply by hand. It is not smooth in trace, and it push a wrong data (issue ?) in polygone widget.

Thank you for the great work you all do on this project. It give us so many possibilities. Long life to Open Data Kit !!

2 Likes

Thanks Hélène for this sythesis.

I made some comments on the google doc.

Mathieu

Mathieu Bossaert
Responsable du Système d'Information
Ligne directe : 04 67 29 90 65

1 Like

Hi RĂ©my,

in fact i don't care if we need 3 GEO widget (point/trace/shape) or only one.
What matters for me is to have clear difference between a point , a line (well formed) and a polygon (well formed).
And that must be clear for users too. Do they draw a point, a line or a polygon.

A generic geowidget as we used in GeoODK is of course more comfortable because we only ask one question in the form.

Thanks for the comments! +1 comments as you have added @mathieubossaert are super helpful because they will help with prioritization. @Remy_CLEMENT if you have other thoughts on the specific changes described in the docs, those would be much appreciated too.

And I also want to be super clear that most of the awesome proposals so far are from @zestyping and @Ivangayton who have been experimenting with some of the changes already. I have not yet been personally involved in building forms or supporting deployments that use traces and shapes so my impressions are limited to issues I've noticed from toy forms. I see geo as a huge area of potential but don't have a ton of knowledge or experience myself.

A generic geowidget that lets the enumerator decide what and how to collect values is an interesting idea. The general ODK model is to structure forms as much as possible to reduce variation in data collected. As I understand it, the goal is to get reasonable data even if enumerators don't understand the problem domain well, have low literacy, low technical experience or aren't trustworthy (seriously, this happens!).

I'm having a hard time imagining a case where any of a point, a line or a shape would satisfy my data needs but I probably just lack imagination! Could you please describe a scenario when you'd like to have a generic geo widget? Also, if it's important to you, it would be helpful to have it in the proposal doc. Please request write permissions and I'll give it to you. My guess is that once we've cleaned up the code and made other improvements it wouldn't be a big deal to introduce a generic widget if there's need for it.

1 Like

I had the opportunity to have a great conversation with @zestyping and @Ivangayton last week to learn about their project and better understand how others could benefit from the work they've done on making the collection of traces easier.

If you'd like to see these improvements in action, the code is at https://github.com/zestyping/collect/tree/ping/geotrace or you can try this debug build: collect-debug-v1.10.1-33-g7cfe90971.apk.zip (1015.8 KB)
(you will need to enable unknown sources if you haven't already and make sure that I am a source you trust).

@Ivangayton and @zestyping iterated on these features for a drainage mapping project and have been using them in their deployment for a little while now (I hear @Ivangayton is working on a blog post and hopefully can share it soon!). They would like to see the improvements in the core so that they don't need to maintain their own Collect version but are not in a rush since their version is working well for them currently.

Next steps are

  • continue getting feedback and agreement on the changes since some of them will require enumerator re-training. This means anyone who uses geotrace and geoshape widgets should follow this thread. @yanokwa, can you please make sure to include this in the next post that goes out to the announcements category?
  • figure out how these changes will fit in with testing and code improvements that @jknightco is doing on the geo widgets.
  • work with @zestyping to figure out how best to pull in changes (in what order, which should be combined, whether they should all be released at once, etc)

@jknightco, I think it makes sense to first complete what you're working on so that there are some tests in place and so that we can more clearly see if and how the improvements could also be applied to the variants with Google Maps basemaps. Does that sound right? Do you have a rough sense of when that might be complete?

Thanks everyone!

1 Like

Good news and bad news:

The bad news is that the current set of tests are all around the functionality of the Widgets while the changes in the zestyping fork are almost exclusively dealing with the Geotrace Activity. The good news is that the changes look relatively straight forward and are a definite improvement on the code quality for the Geotrace Activity.

I would like to see some tests written for GeoPointAggregator as its got quite a bit of computation going on, and I think we should have a brief discussion about the move from defining Views in-code to in-XML (which I think is an improvement but it is a diversion from the current norm nonetheless), but assuming the functionality changes introduce by the fork are an improvement I say we should get them integrated ASAP.

As far as the other changes/improvements go, I'm afraid I have to defer to folks with more hands-on experience with how the app is used. I think the proposal to unite the widgets into a single function is interesting, but I can't say whether that's worth our time or not. At the very least, its definitely a priority to get the current widgets in line with what the XForm specs dictate.

Regarding long term code health, I think the current Geo Activities are definitely one of the messier sections of code. If we want to make larger changes to them, I suggest we go with a rewrite that attempts to unify as much of the base functionality into a single class and then only overrides that functionality when necessary. I'm happy to take this on if everyone thinks it would be worthwhile.

Apologies for the winding post, there's a lot of moving parts here. To recap:

  1. Current batch of tests isn't touching the underlying Geo Activities, just verifying that the Geo Widgets do their job of saving/loading Answers.
  2. zestyping fork looks like a general improvement and should be merged soon assuming functionality changes are OK'd.
  3. zestyping fork could use some tests for GeoPointAggregator.
  4. Other improvements/changes should be prioritized and a decision should be made on a refactor or rewrite based on those priorities.
  5. My vote for top priority is at least getting in line with the XForms spec.
1 Like

I think the tests are good to have in place first because they at least make sure values are being saved as expected. Perhaps we can also think of some tests of the activities that would ensure that any behavior that should stay the same actually does.

The biggest reason I think the refactor should come before pulling in @zestyping's changes is that it will be confusing to have dramatically different behavior and interfaces between the Google Maps and OSM variants of the widgets. I think it makes more sense to think of them as interchangeable.

Agreed that separating trace and shape should be a priority. That said, my sense is that doing all the changes we agree to at once will be less disruptive than rolling them out bit by bit. By grouping them together we can clearly document the new behavior, announce it far and wide and provide a guide for retraining enumerators.

2 Likes

doing all the changes we agree to at once will be less disruptive than rolling them out bit by bit

Could not agree more. If we sell it as a wholesale "we've completely overhauled the geo widgets, brought them in line with the spec, and improved their performance," I think the changes will be seen very positively.

4 Likes

Hi everyone!

Most of the improvements described in this topic and discussion are now completed. We are no longer using the Google document mentioned at the top; the remaining items have been moved to issues in Github:

1 Like

5 posts were split to a new topic: How can I collect a manual geotrace?