This is part of a series of blogs about my journey working on a collaborative Field Mapping tool, now called FieldTM:

pt1 here.

pt2 here.

pt3 here.

pt4 here.

Field Mapping: The Past

Paper Era

• People actually used to have to write things down on paper, remember information in their heads, and talk to one another - damn! However ever did they manage?

• In all seriousness, coordinating a mapping campaign before the 1990’s was probably a logistical nightmare. Paper maps, scrawled notes by field teams that had to be coordinated at the start of the day, then sent on their way.

• No doubt, labour intensive and prone to error.

Digital Transition

• With the advent of handheld GPS devices in the 1990s and early 2000s, digital coordinates could be logged in the field.

• Paired with rugged PDAs or laptops running early GIS software like ArcPad, fieldwork became more accurate and geospatially aligned - but the workflow was still often clunky and required post-field data syncing.

  

… See full entry

See pt1 of this series here.

See pt2 of this series here.

See pt3 of this series here.

How Does This Relate To OSM Again?

If you have been following the articles listed above, you will know the Field Mapping Tasking Manager (FMTM) is a tool developed by HOTOSM to improve the quality of field verified data associated with geometries.

In the OSM world, this means adding and improving tags for OSM geometries.

A key element of this work flow is conflation of newly collected data with existing data in OSM. Some preliminary work was done on this, but for now the team decided to pivot and focus entirely on improving the usability of FMTM from the perspective of mappers.

The conflation work will be continued further down the line.

Currently, Rob Savoye is also continuing some work in parallel for conflation of roads in the USA, with OSMUS’s osm-merge project.

The roadmap for FMTM can be found here.

Easier Field Mapping

Since release 2024.5.0, we have be focusing on what we have dubbed the ‘Mapper Frontend’.

Our primary goals:

• Mapping should be as intuitive as possible for users, requiring minimal training or existing knowledge (with prompts).

• The application should be fast and responsive, tailored to field mapping needs.

• Data should load real-time, significantly improving the collaborative team experience. Joe wants to know where Jim is mapping currently, to avoid overlap.

• A secondary goal is the potential to package it up in a mobile app wrapper, allowing for easy distribution via mobile app stores.

… See full entry

ODK

For those that don’t know, ODK is an incredible suite of tools for field data collection.

The Field Mapping Tasking Manager(FMTM) leverages two of their tools to coordinate field tagging of map data:

• ODK Central as the centralised server to store survey data.
• ODK Collect as the mobile app for survey-based data collection on mobile phones (working very nicely in offline contexts too).

Entities

• ODK Entities were introduced to Central in September 2023, in order to more easily track the same feature over time.
• As a result, we have a nice way to store a feature, with geometry and properties, in ODK Central.
  • This could quite easily map to the OSM ID, feature geometry, and feature tags.
• The geometry can then be selected in ODK Collect survey questions.

How To Use Entities

Create an Entity List

Within ODK Central, a collection of Entities is called an Entity List (or dataset via the API).

… See full entry

See pt1 of this series here.

See pt2 of this series here.

A Long Overdue Release

The goal with the Field Mapping Tasking Manager (FMTM) was to adopt an agile development style, making a new release around once per month.

3 months have passed since the last blog post and there was no FMTM release in between!

What gives??

Well, the team has been working extremely hard simplifying usage of FMTM and making it much more usable.

It’s been hard to find a good point to stabilise a release due to so many great and rapidly developed updates!

From this release onwards we plan to follow through with a new version increment every month. Look out for version 2024.4.0 next month.

Public Beta Now Live

• The main goal we have been working towards is releasing a public beta this month.
• The public beta for FMTM is now live on https://fmtm.hotosm.org
• The idea to to have the public test out it’s functionalities & definitely break a thing or two!
• With your valuable feedback we want to make FMTM the best it can possibly be 🙏

… See full entry

To GDAL or not to GDAL

GDAL is an incredible geospatial library and underpins so much of what we do, including our databases (PostGIS).

However, sometimes it might be a bit heavyweight for what we are trying to achieve.

Installing it as a base system dependency inevitably installs everything - there are no options.

Install size is especially important when building container images, that we want to be as small as possible for distribution.

GDAL in PostGIS

PostGIS uses GDAL for most of it’s geospatial processing, including reading and writing various geospatial file formats.

FMTM is starting to use FlatGeobuf format for various purposes (OSM data extracts, storing submissions).

It also uses a PostGIS database as part of the software stack.

So today I thought: why not just use the geospatial processing built into PostGIS for reading and writing flatgeobuf data?

The solution was surprisingly painless!

Database Access

First we need a way to access the database.

… See full entry

See pt1 of this series here.

First Release

We have been running various versions of FMTM for some time now, but released our first official version not long ago: v0.1.0.

See the excellent release notes written by Susmina Manandhar, product manager for FMTM, for further details of features, bug fixes, and improvements.

Just to reiterate what a wonderful team of devs we have working on the project: https://www.openstreetmap.org/user/varun2948, https://www.openstreetmap.org/user/nrjadkry, https://www.openstreetmap.org/user/Sujanadh, https://www.openstreetmap.org/user/NSUWAL.

Special thanks for https://www.openstreetmap.org/user/JoltCode, an volunteer, who helped to modernise our frontend build tools. And to https://www.openstreetmap.org/user/robsavoye for his work on osm-fieldwork and osm-rawdata that underpin FMTM, plus guidance from the start.

Try it out!

I have been working to ease the installation process for organisations that need to run FMTM.

… See full entry

Storing map tiles in a single file is a common way to load basemaps on a map client.

There are a few formats available to do this, with different use cases.

Offline

mbtiles

• A format innovated by Mapbox, but is a fully open spec.
• Essentially an SQLite database linking to embedded tiled images.
• The client interfaces with the database and loads each tile as required by the basemap.

OSMAnd SQLite

• Based on BigPlanet SQLite format.
• Basically the same as mbtiles, but a slightly different database schema.

A small aside.

Sometimes it’s necessary to generate both mbtiles and OsmAnd format to view in different software, which is a pain.

There is an open issue in OsmAnd to support mbtiles format, but it’s not a priority for now.

Knowing that they are very similar file formats, I considered the possibility of accessing one SQLite database via another ‘wrapper’ SQLite database in a custom view. This view would map tables and fields from one database schema to the other, eliminating the need to store both tilesets for the same data.

Assuming you have an MBTiles table with the following schema: CREATE TABLE mbtiles_table ( zoom_level INTEGER, tile_column INTEGER, tile_row INTEGER, tile_data BLOB );

And you want to create a view for an OsmAnd SQLite table with a schema like: CREATE TABLE osmand_table ( _id INTEGER PRIMARY KEY AUTOINCREMENT, x INTEGER, y INTEGER, z INTEGER, tile_data BLOB );

You can create a view to convert between them like this: CREATE VIEW osmand_mbtiles_view AS SELECT NULL AS _id, -- Use NULL for auto-increment _id tile_column AS x, tile_row AS y, zoom_level AS z, tile_data FROM mbtiles_table;

I couldn’t get this to work when testing, however (it may warrant further investigation).

If you find a solution, please do let me know!

Online

PMTiles

… See full entry

Manual Mapping

Since the inception of the Missing Maps project in 2014, the global community has achieved remarkable progress in digitally mapping communities that were once poorly mapped or entirely unmapped.

Shout-out to the incredible volunteers that contribute to this during regular mapathons🙏

AI-assisted Mapping

HOTOSM’s fAIr project is at the verge of making this a reality. With an emphasis on open-source (ethical, responsible) models, region-specific training data, and iterative feedback, the entire globe will be mapped at record speed.

Watch this space.

… See full entry