Adding Mapbox vector tile basemaps

Thanks @LN - there's lots of pieces in there, most of which @langstonsmith is probably better able to answer than me. I tend to refer to our documentation like https://www.mapbox.com/help/mobile-offline/ and https://www.mapbox.com/android-docs/maps/overview/offline/ but these questions seem deeper than that, so I'll defer to @langstonsmith.

For the question about GeoPackage, I'm not familiar with that system - I'll see if I can learn more from HOT and OGC folks at FOSS4G later this month.

And for the question about caching vs direct transfer. Mapbox Terms of Service only allow for caching directly on a device for offline use, not direct transfer or 'side-loading'. We have created some special permissions for certain use cases, like offline community mapping, but we need each organization to get in touch with us for that special permission. So for the general Mapbox tiles in the ODK Collect app, it would have to rely on caching - if particular projects or organizations needed to use direct transfer they could connect with us.

1 Like

Hello, @Marena. I'm going to be at FOSS4G as well—looking forward to meeting you there! Thanks for offering this contribution and helping us to figure out how the use cases will work.

As things stand in ODK Collect in the moment, for each type of geographic data widget, there are two entirely separate activities, one that uses Google Maps and one that uses OSM. So we've got two GeoPoint activities, two GeoTrace activities, and two GeoShape activities. This means lots of code duplication and inconsistencies in behaviour between the two implementations in each pair. It also means that adding support for another map SDK (such as MapBox) involves writing a third activity of each type, duplicating even more logic.

The work I'm doing right now is trying to improve the situation by defining a common interface across all the map SDKs. Here's that interface definition:

I'm writing two implementations of this interface, one that uses Google Maps and one that uses OSM. The plan is that we'll only need one activity of each type (GeoPoint, GeoTrace, and GeoShape), and the nothing in the activity will be specific to the map SDK.

@langstonsmith, thanks for the work you're doing! The functionality looks great in that animated GIF you posted. I realize that the new MapFragment interface will mean that some significant refactoring would be required in order to integrate your MapBox work, and I'm sorry to be the bringer of bad news. But, we do want and intend to support many options for map SDKs, and I hope it makes sense why we're moving in this direction!

—Ping

No worries @zestyping. Thanks for the update.

It also means that adding support for another map SDK (such as MapBox) involves writing a third activity of each type, duplicating even more logic.

Was almost finished too :wink: No problem at all. I'll hold off on further work until your interface is finished and I can refactor accordingly.

@langstonsmith Yeah, I'm sorry—I feel bad about making you do more work!

We've reached the point where a first draft of the common interface is ready, and has implementations for both Google and OSM that work for the GeoShape activity (see PR 2465). At this point I expect it to be fairly stable, except for the addition of methods to show editable point markers (for GeoPoint) and polylines (for GeoTrace), which will follow the pattern of the existing methods for showing editable polygons.

I'd welcome your feedback on it—would you mind having a look at the MapFragment interface and letting me know if you see anything about its design that would make it difficult to implement with MapBox?

1 Like

I feel bad about making you do more work!

No worries. I learned a ton about ODK and the app! We're making progress :slightly_smiling_face:

I probably won't have time to truly dive into the #2465 pr (and maybe add a Mapbox map fragment) until the 2nd half of next week :unamused: This refactoring will be a good test of the Mapbox Maps SDK's SupportMapFragment !

We recently added draggable markers, so I need to figure out how that might fit into ODK.

I think I'm going to keep waiting to see what you end up with and then see how Mapbox fits into it all

Hi all, Paul from Humanitarian OpenStreetMap team (HOT) here. Access to and interfaces for using vector tiles sounds like a great feature to have! For accessibility and ease of use, having a default Mapbox tiles option would be a great place to start.

I think that ideally, integrating vector tiles would contribute to, and work towards more effective geospatial data collection in ODK across the board, and not be a parallel effort limited to use as a 'plain' basemap. I fully understand there are tradeoffs involved on Mapbox's end, but some of the complications/limitations I see are that Mapbox tiles limit the features they contain and expose, it'll probably be complicated to access to underlying OSM id's (and might lack the ability to attach extra data/attributes from form questions), and that portablity is potentially limited.

Some questions/suggestions from my end (these do mirror @LN, @yanokwa, and @danbjoseph's to some extent):

  • Would it be feasible to define a "map view" that can use configurable vector and raster sources? And could we define a vector tile interface, for which Mapbox is one of the (default) providers? For example, for cases where the Mapbox tiles don't suffice, we could then also have the option to package vector tile data for deployments, or set up vector tile endpoints containing a specific AOI and data types we need (say, all WaSH facilities + attributes we need for those).
  • As @Marena indicated, it would be awesome if the vector tiles would be stylable via MapCSS, or another (portable/standardized) format!
  • As @danbjoseph requested, designing to allow offline usage would be a major benefit. ODK is used by many humanitarian organizations in low (or non-existant) bandwidth environments. Caching is one thing, but requires those specific devices to be connected at some point. In a lot of our projects we need to be able to package and preload all map data in advance (aerial and raster basemaps as mbtiles, editable map data as .osm) before embarking.
  • What options do you see if we were not just to think about displaying basemaps, but also enhancing editing and geospatial data collection? Would that somehow be feasible on vector tiles? What (standard) file formats would be the most suitable for that? OpenMapKit has been using .osm xml, but I'd like to see us move towards open (OGC) formats if possible, like GeoPackage. Thoughts?
1 Like

Hi!

First, an update on the code. @langstonsmith, the new MapFragment interface is now on the master branch, and it is in use by the newly consolidated GeoTraceActivity and GeoShapeActivity. So I think it is mature enough to be ready for new implementations. There are existing Google Maps and OSM implementations (GoogleMapFragment and OsmMapFragment) that you can use as examples if you'd like to start building a MapboxMapFragment implementation.

Note that the configuration settings to select the basemap are handled by MapHelper, which has not yet been refactored. My plan is to move the functionality of MapHelper into each of the appropriate MapFragment implementations, so my advice to you is to keep your changes to MapHelper minimal (e.g. just hardcode a vector tile source for now) and focus on implementing MapFragment. Feel free to reach out to me any time if you have questions.

Second, I just want to clarify the question of online vs. offline tiles. @langstonsmith, is your plan to implement online vector tiles only, or will there be support for both online and offline vector tiles? What would be involved in making it possible to display offline vector tiles that are packaged and preloaded onto the mobile device's filesystem in advance, as @paul.uithol describes in his comment?

@paul.uithol, can you elaborate a little on what you meant in your first bullet point, "defining a 'map view' that can use configurable vector and raster sources"? At the moment, ODK Collect already has configuration settings such that:

  • You can choose the Google map or OSM map
  • You can choose which standard basemap style you want (e.g. satellite, terrain, etc.)
  • You can choose a raster tile file from the device's filesystem (an mbtiles file)

If @langstonsmith then adds a Mapbox implementation, we'd have a third option in addition to Google and OSM, and presumably there would be some configuration setting for choosing the vector tile set to use as the basemap.

Does that meet your requirements, or can you help me understand how what you have in mind would be different from this?

Thanks!

3 Likes

Awesome. Amazing update @zestyping. Thanks.

I will try to look at the master branch in the next couple of days and see what can be done to add Mapbox. My mind is definitely thinking about online tiles as a start.

Using Mapbox offline vector tiles largely revolves around declaring and downloading certain desired region(s). The region could be declared ahead of time or chosen by a user. The Offline Plugin for Android helps with this.

Anyways. More to come.

2 Likes

Hi @langstonsmith, how's it going?

@paul.uithol, any further thoughts on the questions in my last comment above?

@zestyping, busy couple of weeks of work travel. I re-based my branch with master to pull in the #2465 refactor and all of the other changes. Lots of conflicts but I've cleaned them up at this point :slightly_smiling_face:

There are existing Google Maps and OSM implementations (GoogleMapFragment and OsmMapFragment) that you can use as examples if you'd like to start building a MapboxMapFragment implementation

so my advice to you is to keep your changes to MapHelper minimal (e.g. just hardcode a vector tile source for now) and focus on implementing MapFragment

The Mapbox Maps SDK already has a SupportMapFragment :muscle: The state of my pr is that I've created a MapboxMapFragment that:

  • extends SupportMapFragment
  • implements the MapFragment interface

Many more tweaks are needed so that data is shown properly on a Mapbox map, depending on the specific widget. But yea. My pr now has your changes and things still look possible/promising.

I'm still not at a point where I want to open the pull request, but here's a link to a collection of GIFs which show the current state of my work. Vector tiles are working well :ok_hand:

I'm still looking into how offline can be supported via Mapbox and what the Mapbox Maps SDK can handle (related ticket). Seems that the current MBTile format that Mapbox spits out, still can't be processed by the app. I'm planning to ask some colleagues some clarifying questions, because this is new territory for me.

In my opinion, some things that have come up in this thread, such as some of @paul.uithol 's suggestions, are future feature requests, rather than blockers for my specific additions.

I will definitely have Marena continue running point on Mapbox access token and what makes sense for the ODK community.

@zestyping the mapping interfaces are great! Based on map UI interaction and device location listeners, it seems that I've hooked Mapbox into everything correctly.

Happy to keep the questions and conversations going here, but I'm feeling like I'm coming into the homestretch of the pr. Continued offline investigations, checkstyle cleanup, and testing, seem like the final 3 remaining things.

4 Likes

Wow, these screen video GIFs look fantastic! Beautiful work!

I'm pleased to hear that it seems like the MapFragment interface is working out for you—did you have to make any changes to get it to play well with your implementation classes?

I also noticed GeoPoint among your GIFs, which is the one class I hadn't yet migrated to use the new MapFragment interface. (I have some work in progress on that, but haven't pushed that branch yet.) How did you do that—did you do your own migration of GeoPoint to use the new interface?

Not many. Making tweaks based on an already-working copy of the GoogleMapFragment was a great way to start :slightly_smiling_face: As I said above, the Mapbox Maps SDK already has a SupportMapFragment, so most of the work was adding in our own LocationEngine stuff. Other work was doing simple package name swapping. For example, swapping the Google Maps Polyline class for the Mapbox Polyline class/package.

I also noticed GeoPoint among your GIFs, which is the one class I hadn't yet migrated to use the new MapFragment interface. (I have some work in progress on that, but haven't pushed that branch yet.) How did you do that—did you do your own migration of GeoPoint to use the new interface?

Yea, I just copied the Google Maps GeoPointMapActivity and made adjustments for a Mapbox equivalent. There aren't a ton of methods, so it didn't take very long.

1 Like

Hi everyone - very exciting to see how this is moving! I have a proposal for how to handle the question of access tokens.

For the official ODK Collect app (available in the Play Store)

  • We set up an ODK Mapbox account with @yanokwa @zestyping and use the token from that account for the Play Store app. This token isn't publicly published to Github and won't carry over to any clone/fork of the ODK Collect repo. It is only for the Play Store app.
  • All the official ODK Collect app traffic goes on one account that Mapbox can support with an annual coupon (or special plan if need be), akin to Google's approach of supporting ODK's use of their APIs.

For forks of ODK Collect

  • Devs are asked to create their own Mapbox account/token to use in their fork
  • We include instructions on how to contact the Mapbox Community team if the fork is for a non-profit or the Mapbox Sales team if for a commercial project with high volume.
  • I'm thinking we add something like the following to ODK Collect's developer page:

Mapbox APIs - If you want to use Mapbox services within your fork of ODK Collect, you will need to create a Mapbox account in order to have a Mapbox access token. Opening a Mapbox account on the 'Pay-As-You-Go' plan does not require a credit card. Mapbox provides free API usage up to the monthly thresholds documented at https://www.mapbox.com/pricing/ (e.g. below 50,000 monthly active users is free). If your usage exceeds any of these thresholds, you will receive an automatic email with instructions on how to add a credit card to pay for the services. Services will remain live until the end of the 30-day billing term, after which point the account will be deactivated and requires a credit card to reactivate.

Once you have created a Mapbox account, find your access token as described here and paste your access token in --- [instructions to be defined by @langstonsmith ]---. Please be sure not to commit your personal access token to a branch that you will submit a pull request for.

Learn more about Mapbox access tokens here and Mapbox terms of service here.

If you are creating a fork of ODK Collect for a non-profit organization or positive impact project, you can contact the Mapbox Community team with any questions about using Mapbox. If you are creating a fork of ODK Collect for a commercial enterprise and have questions about pricing or terms, you can contact the Mapbox Sales team.

Does anyone have concerns about this approach?

Great seeing the updates and progress here @zestyping @langstonsmith @Marena.

@Marena Agreed with the proposal for having a single account for the official ODK Collect app and then giving dev's options for fork's or working with a fork of ODK Collect. I don't see any issues with this approach at the moment.

@zestyping Following up to your question about @paul.uithol's comment. Yes, I think that's in line with what he was talking about. Being able to configure based on different online or offline options is something we'd be looking for. Some of the other items, like editing, sound like are beyond the scope of this thread and are starting to get towards this new thread: Load existing data points in to a geopoint widget in Collect

Haven't forgotten about this. @zestyping, it seems as though you continue to make great progress on map-related work in the app: https://github.com/opendatakit/collect/pulls/zestyping

I'm thinking about holding off on trying to finish my own work until your major refactoring is complete...

Hey @langstonsmith!

I'd say we're done with the major refactoring; the MapFragment interface is pretty stable now. Of the pull requests you see there, only #2776 touches MapFragment; the rest are behaviour changes above the MapFragment level, which should not affect your work.

Here's the latest MapFragment interface, which is just pending QA; I expect it to be on master by next week: https://github.com/opendatakit/collect/blob/568823acbc2a6027863063aa09cb4b988bc38424/collect_app/src/main/java/org/odk/collect/android/map/MapFragment.java

Can you help me understand what steps you have left to do to finish your work? If there are any concerns that are holding you back, I'd like to address them promptly so you can proceed.

1 Like

Ah, I see now @zestyping . Your other prs branch off of #2776 :slightly_smiling_face:

Still wrestling with some funky Fragment lifecycle stuff on the Mapbox side. Will also kinda' wait until #2776 lands so that I can pull the changes into my branch.

1 Like

There's a new related discussion here: Geo: Using the Mapbox SDK for Android

Hey, thanks for Geo: Using the Mapbox SDK for Android .

I'm reeeeealllly close to opening my pr. Here's my work https://github.com/opendatakit/collect/compare/master...langsmith:ls-adding-mapbox-maps?expand=1

I need to finish hooking the Mapbox Annotation Plugin for Android into some of the logic related to marker clearing, line dragging, etc.

Adding Mapbox vector tile basemaps wasn't mentioned in today's Geo: Using the Mapbox SDK for Android ticket. Might be good to cross reference them.

1 Like