A major version of uMap has been released last week.

This release is inauguring a new era in versioning uMap: in the future, we’ll take care of better documenting breaking changes, so expect more major releases from now on. More details on how we version.

A comprehensive changelog for that version is available in our technical documentation. Most of the changes for a major version are indeed technical, we are taking care of people deploying and maintaining instances with that approach. User-facing features are deployed continuously with our minor versions. We think that scheme is more valuable for the community.

It doesn’t mean you will not see improvements with that release though. Most notably, you’ll face:

A revamped dashboard with multiple options for your maps (including a bulk download / backup)

A hard work by @davidbgk and @Aurelie-Jallut!

Experimental drag and drop of file on the map container

Add minimal CSV export

This sounds like this export should have existed before, but until now we did not decide what to do with complex geometries, and now we did: for now it just add the center. Let’s use it and see if we need something more complex.

Highlight selected feature, finally!

A contribution of @jschleic, with the help of @Aurelie-Jallut!

Full map download endpoint

You can now backup your map programmatically! The URL should look like https://yourumapdoma.in/map/{yourid}/download/

Refactor icon image selector

Very minimal optimistic conflicts resolution mechanism

Thanks to @almet for this, and to @Biondilbiondo for the initial idea and implementation

See also the work in progress (🚧 meaning not yet in production) by @almet to make live collaborative editing a thing in uMap.

New tilelayer switcher (thanks to Leaflet.IconLayers)

Add counter in datalayer headline

Allow to type a latlng in the search box

Add a popup template to showcase OpenStreetMap data

The idea is to easily show the most common tags in OSM (name, email, address…). It’s very basic for now, so let’s make it better together!

Refactor Share & Download UI for better usability

Thanks to @jschleic for this contribution!

And even more

• an attention to accessibility and contrasts (more to come!)
• the ability to use oEmbed

Thank you note

Since May 2023, uMap has received support from 2 French government agencies: Agence Nationale de la Cohésion des Territoires and Accélérateur d’initiatives citoyennes. Also, a NGI Grant from the NLNet foundation is in progress to work on the collaborative editing feature. A dedicated instance has been up for a couple month already useful to close to a hundred public agents. This has allowed to finance a team of 7 to work part-time on the project. We’ve recently received confirmation that the funding would be continued in 2024 so keep expecting a lot of activity on the project!

The team is still dedicated to make uMap more stable, accessible and we welcome newcomers so do not hesitate to:

• join us on Matrix,
• contribute to the code,
• and/or give us your feedback through the mailing-list (en) or the forum (fr).

Also don’t miss out on the revamped developer docs website and our the brand new official uMap Project website.

If you appreciate our work and want to support the team behind it please consider donating on:

• Libera Pay
• Open Collective

The uMap team

I’ve spent the last few weeks working on uMap, still with the goal of bringing real-time collaboration to the maps. I’m not there yet, but I’ve made some progress that I will relate here.

JavaScript modules

uMap has been there since 2012, at a time when ES6 wasn’t out there yet

At the time, it wasn’t possible to use JavaScript modules, nor modern JavaScript syntax. The project stayed with these requirements for a long time, in order to support people with old browsers. But as time goes on, we can now have access to more features.

The team has been working hard on bringing modules to the mix, and it wasn’t a piece of cake. But, the result is here: we’re now able to use modern JavaSript modules and we are now more confident about which features of the languages we can use or not

I then spent ~way too much~ some time trying to integrate existing CRDTs like Automerge and YJS in our project. These two libs are unfortunately expecting us to use a bundler, which we aren’t currently.

uMap is plain old JavaScript. It’s not using react or any other framework. The way I see this is that it makes it possible for us to have something “close to the metal”, if that means anything when it comes to web development. We’re not tied to the development pace of these frameworks, and have more control on what we do. It’s easier to debug.

So, after making tweaks and learning how “modules”, “requires” and “bundling” was working, I ultimately decided to take a break from this path, to work on the wiring with uMap. After all, CRDTs might not even be the way forward for us.

Internals

I was not expecting this to be easy and was a bit afraid. Mostly because I’m out of my comfort zone. After some time with the head under the water, I’m now able to better understand the big picture, and I’m not getting lost in the details like I was at first.

Let me try to summarize what I’ve learned.

uMap appears to be doing a lot of different things, but in the end it’s:

• Using Leaflet.js to render features on the map ;
• Using Leaflet Editable to edit complex shapes, like polylines, polygons, and to draw markers ;
• Using the Formbuilder to expose a way for the users to edit the features, and the data of the map
• Serializing the layers to and from GeoJSON. That’s what’s being sent to and received from the server.
• Providing different layer types (marker cluster, chloropleth, etc) to display the data in different ways.

Naming matters

There is some naming overlap between the different projects we’re using, and it’s important to have these small clarifications in mind:

Leaflet layers and uMap features

In Leaflet, everything is a layer. What we call features in geoJSON are leaflet layers, and even a (uMap) layer is a layer. We need to be extra careful what are our inputs and outputs in this context.

We actually have different layers concepts: the datalayer and the different kind of layers (chloropleth, marker cluster, etc). A datalayer, is (as you can guess) where the data is stored. It’s what uMap serializes. It contains the features (with their properties). But that’s the trick: these features are named layers by Leaflet.

GeoJSON and Leaflet

We’re using GeoJSON to share data with the server, but we’re using Leaflet internally. And these two have different way of naming things.

The different geometries are named differently (a leaflet Marker is a GeoJSON Point), and their coordinates are stored differently: Leaflet stores lat, long where GeoJSON stores long, lat. Not a big deal, but it’s a good thing to know.

Leaflet stores data in options, where GeoJSON stores it in properties.

This is not reactive programming

I was expecting the frontend to be organised similarly to Elm apps (or React apps): a global state and a data flow (a la redux), with events changing the data that will trigger a rerendering of the interface.

Things work differently for us: different components can write to the map, and get updated without being centralized. It’s just a different paradigm.

A syncing proof of concept

With that in mind, I started thinking about a simple way to implement syncing.

I left aside all the thinking about how this would relate with CRDTs. It can be useful, but later. For now, I “just” want to synchronize two maps. I want a proof of concept to do informed decisions.

Syncing map properties

I started syncing map properties. Things like the name of the map, the default color and type of the marker, the description, the default zoom level, etc.

All of these are handled by “the formbuilder”. You pass it an object, a list of properties and a callback to call when an update happens, and it will build for you form inputs.

Taken from the documentation (and simplified):

var tilelayerFields = [
    ['name', {handler: 'BlurInput', placeholder: 'display name'}],
    ['maxZoom', {handler: 'BlurIntInput', placeholder: 'max zoom'}],
    ['minZoom', {handler: 'BlurIntInput', placeholder: 'min zoom'}],
    ['attribution', {handler: 'BlurInput', placeholder: 'attribution'}],
    ['tms', {handler: 'CheckBox', helpText: 'TMS format'}]
];
var builder = new L.FormBuilder(myObject, tilelayerFields, {
    callback: myCallback,
    callbackContext: this
});

In uMap, the formbuilder is used for every form you see on the right panel. Map properties are stored in the map object.

We want two different clients work together. When one changes the value of a property, the other client needs to be updated, and update its interface.

I’ve started by creating a mapping of property names to rerender-methods, and added a method renderProperties(properties) which updates the interface, depending on the properties passed to it.

We now have two important things:

1. Some code getting called each time a property is changed ;
2. A way to refresh the right interface when a property is changed.

In other words, from one client we can send the message to the other client, which will be able to rerender itself.

Looks like a plan.

Websockets

We need a way for the data to go from one side to the other. The easiest way is probably websockets.

Here is a simple code which will relay messages from one websocket to the other connected clients. It’s not the final code, it’s just for demo puposes.

A basic way to do this on the server side is to use python’s websockets library.

import asyncio
import websockets
from websockets.server import serve
import json

# Just relay all messages to other connected peers for now
CONNECTIONS = set()
async def join_and_listen(websocket):
CONNECTIONS.add(websocket)
try:
async for message in websocket:
# recompute the peers-list at the time of message-sending.
# doing so beforehand would miss new connections
peers = CONNECTIONS - {websocket}
websockets.broadcast(peers, message)
finally:
CONNECTIONS.remove(websocket)
async def handler(websocket):
message = await websocket.recv()
event = json.loads(message)
<span class="c1"># The first event should always be 'join'</span>
<span class="k">assert</span> <span class="n">event</span><span class="p">[</span><span class="s2">"kind"</span><span class="p">]</span> <span class="o">==</span> <span class="s2">"join"</span>
<span class="k">await</span> <span class="n">join_and_listen</span><span class="p">(</span><span class="n">websocket</span><span class="p">)</span>
async def main():
async with serve(handler, "localhost", 8001):
await asyncio.Future()  # run forever
asyncio.run(main())

On the client side, it’s fairly easy as well. I won’t even cover it here.

We now have a way to send data from one client to the other. Let’s consider the actions we do as “verbs”. For now, we’re just updating properties values, so we just need the update verb.

Code architecture

We need different parts:

• the transport, which connects to the websockets, sends and receives messages.
• the message sender to relat local messages to the other party.
• the message receiver that’s being called each time we receive a message.
• the sync engine which glues everything together
• Different updaters, which knows how to apply received messages, the goal being to update the interface in the end.

When receiving a message it will be routed to the correct updater, which will know what to do with it.

In our case, its fairly simple: when updating the name property, we send a message with name and value. We also need to send along some additional info: the subject.

In our case, it’s map because we’re updating map properties.

When initializing the map, we’re initializing the SyncEngine, like this:

// inside the map
let syncEngine = new umap.SyncEngine(this)

// Then, when we need to send data to the other party
let syncEngine = this.obj.getSyncEngine()
let subject = this.obj.getSyncSubject()
syncEngine.update(subject, field, value)

The code on the other side of the wire is simple enough: when you receive the message, change the data and rerender the properties:

this.updateObjectValue(this.map, key, value)
this.map.renderProperties(key)

Syncing features

At this stage I was able to sync the properties of the map. A small victory, but not the end of the trip.

The next step was to add syncing for features: markers, polygon and polylines, alongside their properties.

All of these features have a uMap class representation (which extends Leaflets ones). All of them share some code in the FeatureMixin class.

That seems a good place to do the changes.

I did a few changes:

• Each feature now has an identifier, so clients know they’re talking about the same thing. This identifier is also stored in the database when saved.
• I’ve added an upsert verb, because we don’t have any way (from the interface) to make a distinction between the creation of a new feature and its modification. The way we intercept the creation of a feature (or its update) is to use Leaflet Editable’s editable:drawing:commit event. We just have to listen to it and then send the appropriate messages !

After some giggling around (ah, everybody wants to create a new protocol !) I went with reusing GeoJSON. It allowed me to have a better understanding of how Leaflet is using latlongs. That’s a multi-dimensional array, with variable width, depending on the type of geometry and the number of shapes in each of these.

Clearly not something I want to redo, so I’m now reusing some Leaflet code, which handles this serialization for me.

I’m now able to sync different types of features with their properties.

Point properties are also editable, using the already-existing table editor. I was expecting this to require some work but it’s just working without more changes.

What’s next ?

I’m able to sync map properties, features and their properties, but I’m not yet syncing layers. That’s the next step! I also plan to make some pull requests with the interesting bits I’m sure will go in the final implementation:

• Adding ids to features, so we have a way to refer to them.
• Having a way to map properties with how they render the interface, the renderProperties bits.

When this demo will be working, I’ll probably spend some time updating it with the latest changes (umap is moving a lot these weeks). I will probably focus on how to integrate websockets in the server side, and then will see how to leverage (maybe) some magic from CRDTs, if we need it.

See you for the next update!

I continued working on uMap, an open-source map-making tool to create and share customizable maps, based on Open Street Map data.

Here is a summary of what I did:

• I reviewed, rebased and made some minor changes to a pull request which makes it possible to merge geojson features together ;
• I’ve explored around the idea of using SQLite inside the browser, for two reasons : it could make it possible to use the Spatialite extension, and it might help us to implement a CRDT with cr-sqlite ;
• I learned a lot about the SIG field. This is a wide ecosystem with lots of moving parts, which I understand a bit better now.

The optimistic-merge approach

There were an open pull request implementing an “optimistic merge”. We spent some time together with Yohan to understand what the pull request is doing, discuss it and made a few changes.

Here’s the logic of the changes:

1. On the server-side, we detect if we have a conflict between the incoming changes and what’s stored on the server (is the last document save fresher than the IF-UNMODIFIED-SINCE header we get ?) ;
2. In case of conflict, find back the reference document in the history (let’s name this the “local reference”) ;
3. Merge the 3 documents together, that is :
4. Find what the the incoming changes are, by comparing the incoming doc to the local reference.
5. Re-apply the changes on top of the latest doc.

One could compare this logic to what happens when you do a git rebase. Here is some pseudo-code:

def merge_features(reference: list, latest: list, incoming: list):
    """Finds the changes between reference and incoming, and reapplies them on top of latest."""
    if latest == incoming:
        return latest

<span class="n">reference_removed</span><span class="p">,</span> <span class="n">incoming_added</span> <span class="o">=</span> <span class="n">get_difference</span><span class="p">(</span><span class="n">reference</span><span class="p">,</span> <span class="n">incoming</span><span class="p">)</span>

<span class="c1"># Ensure that items changed in the reference weren't also changed in the latest.</span>
<span class="k">for</span> <span class="n">removed</span> <span class="ow">in</span> <span class="n">reference_removed</span><span class="p">:</span>
    <span class="k">if</span> <span class="n">removed</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">latest</span><span class="p">:</span>
        <span class="k">raise</span> <span class="n">ConflictError</span>

<span class="n">merged</span> <span class="o">=</span> <span class="n">copy</span><span class="p">(</span><span class="n">latest</span><span class="p">)</span>
<span class="c1"># Reapply the changes on top of the latest.</span>
<span class="k">for</span> <span class="n">removed</span> <span class="ow">in</span> <span class="n">reference_removed</span><span class="p">:</span>
    <span class="n">merged</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">removed</span><span class="p">)</span>

<span class="k">for</span> <span class="n">added</span> <span class="ow">in</span> <span class="n">incoming_added</span><span class="p">:</span>
    <span class="n">merged</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">added</span><span class="p">)</span>

<span class="k">return</span> <span class="n">merged</span>

The pull request is not ready yet, as I still want to add tests with real data, and enhance the naming, but that’s a step in the right direction :-)

Using SQLite in the browser

At the moment, (almost) everything is stored on the server side as GeoJSON files. They are simple to use, to read and to write, and having them on the storage makes it easy to handle multiple revisions.

I’ve been asked to challenge this idea for a moment. What if we were using some other technology to store the data? What would that give us? What would be the challenges?

I went with SQLite, just to see what this would mean.

• SQLite is originally not made to work on a web browser, but thanks to Web Assembly, it’s possible to use it. It’s not that big, but the library weights 2Mb.
• With projects such as CR-SQLite, you get a way to add CRDTs on top of SQLite databases. Meaning that the clients could send their changes to other clients or to the server, and that it would be easy to integrate ;
• The clients could retrieve just some part of the data to the server (e.g. by specifying a bounding box), which gives it the possibility to not load everything in memory if that’s not needed.

I wanted to see how it would work, and what would be the challenges around this technology. I wrote a small application with it. Turns out writing to a local in-browser SQLite works.

Here is what it would look like:

• Each client will get a copy of the database, alongside a version ;
• When clients send changes, you can just send the data since the last version ;
• Databases can be merged without loosing data, the operations done in SQL will trigger writes to a specific table, which will be used as a CRDT.

I’m not sure SQLite by itself is useful here. It sure is fun, but I don’t see what we get in comparison with a more classical CRDT approach, except complexity. The technology is still quite young and rough to the edges, and uses Rust and WebASM, which are still strange beasts to me.

Related projects in the SIG field

Here are some interesting projects I’ve found this week :

• Leaflet.offline allows to store the tiles offline ;
• geojson-vt uses the concept of “vector tiles” I didn’t know about. Tiles can return binary or vectorial data, which can be useful to just get the data in one specific bounding box This allows us for instance to store GeoJSON in vector tiles.
• mapbox-gl-js makes it possible to render SIG-related data using WebGL (no connection with Leaflet)
• leaflet-ugeojson and leaflet.Sync allows multiple people to share the same view on a map.

Two libraries seems useful for us:

• Leaflet-GeoSSE makes it possible to use SSE (Server Sent Events) to update local data. It uses events (create, update, delete) and keys in the GeoJSON features..
• Leaflet Realtime does something a bit similar, but doesn’t take care of the transport. It’s meant to be used to track remote elements (a GPS tracker for instance)

I’m noting that:

• In the two libraries, unique identifiers are added to the features to allow for updates.
• None of these libraries makes it possible to track local changes. That’s what’s left to find.

How to transport the data?

One of the related subjects is transportation of the data between the client and the server. When we’ll get the local changes, we’ll need to find a way to send this data to the other clients, and ultimately to the server.

There are multiple ways to do this, and I spent some time trying to figure out the pros and cons of each approach. Here is a list:

• WebRTC, the P2P approach. You let the clients talk to each other. I’m not sure where the server fits in this scenario. I’ve yet to figure-out how this works out in real-case scenarii, where you’re working behind a NAT, for instance. Also, what’s the requirement on STUN / Turn servers, etc.
• Using WebSockets seems nice at the first glance, but I’m concerned about the resources this could take on the server. The requirement we have on “real-time” is not that big (e.g. if it’s not immediate, it’s okay).
• Using Server Sent Events is another way to solve this, it seems lighter on the client and on the server. The server still needs to keep connexion opens, but I’ve found some proxies which will do that for you, so it would be something to put between the uMap server and the HTTP server.
• Polling means less connexion open, but also that the server will need to keep track of the messages the clients have to get. It’s easily solvable with a Redis queue for instance.

All of these scenarii are possible, and each of them has pros and cons. I’ll be working on a document this week to better understand what’s hidden behind each of these, so we can ultimately make a choice.

Server-Sent Events (SSE)

Here are some notes about SSE. I’ve learned that:

• SSE makes it so that server connections never ends (so it consumes a process?)
• There is a library in Django for this, named django-eventstream
• Django channels aims at using ASGI for certain parts of the app.
• You don’t have to handle all this in Django. It’s possible to delegate it to pushpin, a proxy, using django-grip

It’s questioning me in terms of infrastructure changes.

Last week, I’ve been lucky to start working on uMap, an open-source map-making tool to create and share customizable maps, based on Open Street Map data.

My goal is to add real-time collaboration to uMap, but we first want to be sure to understand the issue correctly. There are multiple ways to solve this, so one part of the journey is to understand the problem properly (then, we’ll be able to chose the right path forward).

Part of the work is documenting it, so expect to see some blog posts around this in the future.

Installation

I’ve started by installing uMap on my machine, made it work and read the codebase. uMap is written in Python and Django, and using old school Javascript, specifically using the Leaflet library for SIG-related interface.

Installing uMap was simple. On a mac:

1. Create the venv and activate it

python3 -m venv venv
source venv/bin/activate
pip install -e .

1. Install the deps : brew install postgis (this will take some time to complete)

createuser umap
createdb umap -O umap
psql umap -c "CREATE EXTENSION postgis"

1. Copy the default config with cp umap/settings/local.py.sample umap.conf

# Copy the default config to umap.conf
cp umap/settings/local.py.sample umap.conf
export UMAP_SETTINGS=~/dev/umap/umap.conf
make install
make installjs
make vendors
umap migrate
umap runserver

And you’re done!

On Arch Linux, I had to do some changes, but all in all it was simple:

createuser umap -U postgres
createdb umap -O umap -U postgres
psql umap -c "CREATE EXTENSION postgis" -Upostgres

Depending on your installation, you might need to change the USER that connects the database.

The configuration could look like this:

DATABASES = {
    "default": {
        "ENGINE": "django.contrib.gis.db.backends.postgis",
        "NAME": "umap",
        "USER": "postgres",
    }
}

How it’s currently working

With everything working on my machine, I took some time to read the code and understand the current code base.

Here are my findings :

• uMap is currently using a classical client/server architecture where :
• The server is here mainly to handle access rights, store the data and send it over to the clients.
• The actual rendering and modifications of the map are directly done in JavaScript, on the clients.

The data is split in multiple layers. At the time of writing, concurrent writes to the same layers are not possible, as one edit would potentially overwrite the other. It’s possible to have concurrent edits on different layers, though.

When a change occurs, each DataLayer is sent by the client to the server.

• The data is updated on the server.
• If the data has been modified by another client, an HTTP 422 (Unprocessable Entity) status is returned, which makes it possible to detect conflicts. The users are prompted about it, and asked if they want to overwrite the changes.
• The files are stored as geojson files on the server as {datalayer.pk}_{timestamp}.geojson. A history of the last changes is preserved (The default settings preserves the last 10 revisions).
• The data is stored in a Leaflet object and backups are made manually (it does not seem that changes are saved automatically).

Data

Each layer consists of:

• On one side are the properties (matching the _umap_options), and on the other, the geojson data (the Features key).
• Each feature is composed of three keys:
• geometry: the actual geo object
• properties: the data associated with it
• style: just styling information which goes with it, if any.

Real-time collaboration : the different approaches

Behind the “real-time collaboration” name, we have :

1. The streaming of the changes to the clients: when you’re working with other persons on the same map, you can see their edits at the moment they happen.
2. The ability to handle concurrent changes: some changes can happen on the same data concurrently. In such a case, we need to merge them together and be able to
3. Offline editing: in some cases, one needs to map data but doesn’t have access to a network. Changes happen on a local device and is then synced with other devices / the server ;

Keep in mind these notes are just food for toughs, and that other approaches might be discovered on the way

I’ve tried to come up with the different approaches I can follow in order to add the collaboration features we want.

• JSON Patch and JSON Merge Patch: Two specifications by the IETF which define a format for generating and using diffs on json files. In this scenario, we could send the diffs from the clients to the server, and let it merge everything.
• Using CRDTs: Conflict-Free Resolution Data Types are one of the other options we have lying around. The technology has been used mainly to solve concurrent editing on text documents (like etherpad-lite), but should work fine on trees.

JSON Patch and JSON Merge Patch

I’ve stumbled on two IETF specifications for JSON Patch and JSON Merge Patch which respectively define how JSON diffs could be defined and applied.

There are multiple libraries for this, and at least one for Python, Rust and JS.

It’s even supported by the Redis database, which might come handy in case we want to stream the changes with it.

If you’re making edits to the map without changing all the data all the time, it’s possible to generate diffs. For instance, let’s take this simplified data (it’s not valid geojson, but it should be enough for testing):

source.json

{
    "features": [
        {
            "key": "value"
        }
    ],
    "not_changed": "whatever"
}

And now let’s add a new object right after the first one :

destination.geojson

{
    "features": [
        {
            "key": "value"
        },
        {
            "key": "another-value"
        }
    ],
    "not_changed": "whatever"
}

If we generate a diff:

pipx install json-merge-patch
json-merge-patch create-patch source.json destination.json
{
    "features": [
        {
            "key": "value"
        },
        {
            "key": "another-value"
        }
    ]
}

Multiple things to note here:

1. It’s a valid JSON object
2. It doesn’t reproduce the not_changed key
3. But… I was expecting to see only the new item to show up. Instead, we are getting two items here, because it’s replacing the “features” key with everything inside.

This is actually what the specification defines:

4.1. add

The “add” operation performs one of the following functions, depending upon what the target location references:

o If the target location specifies an array index, a new value is inserted into the array at the specified index.

o If the target location specifies an object member that does not already exist, a new member is added to the object

o If the target location specifies an object member that does exist, that member’s value is replaced.

It seems too bad for us, as this will happen each time a new feature is added to the feature collection.

It’s not working out of the box, but we could probably hack something together by having all features defined by a unique id, and send this to the server. We wouldn’t be using vanilla geojson files though, but adding some complexity on top of it.

At this point, I’ve left this here and went to experiment with the other ideas. After all, the goal here is not (yet) to have something functional, but to clarify how the different options would play off.

Using CRDTs

I’ve had a look at the two main CRDTs implementation that seem to get traction these days : Automerge and Yjs.

I’ve first tried to make Automerge work with Python, but the Automerge-py repository is outdated now and won’t build. I realized at this point that we might not even need a python implementation:

In this scenario, the server could just stream the changes from one client to the other, and the CRDT will guarantee that the structures will be similar on both clients. It’s handy because it means we won’t have to implement the CRDT logic on the server side.

Let’s do some JavaScript, then. A simple Leaflet map would look like this:

import L from 'leaflet';
import 'leaflet/dist/leaflet.css';

// Initialize the map and set its view to our chosen geographical coordinates and a zoom level:
const map = L.map('map').setView([48.1173, -1.6778], 13);
// Add a tile layer to add to our map, in this case using Open Street Map
L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', {
    maxZoom: 19,
    attribution: '© OpenStreetMap contributors'
}).addTo(map);
// Initialize a GeoJSON layer and add it to the map
const geojsonFeature = {
    "type": "Feature",
    "properties": {
        "name": "Initial Feature",
        "popupContent": "This is where the journey begins!"
    },
    "geometry": {
        "type": "Point",
        "coordinates": [-0.09, 51.505]
    }
};
const geojsonLayer = L.geoJSON(geojsonFeature, {
    onEachFeature: function (feature, layer) {
        if (feature.properties && feature.properties.popupContent) {
            layer.bindPopup(feature.properties.popupContent);
        }
    }
}).addTo(map);
// Add new features to the map with a click
function onMapClick(e) {
    const newFeature = {
        "type": "Feature",
        "properties": {
            "name": "New Feature",
            "popupContent": "You clicked the map at " + e.latlng.toString()
        },
        "geometry": {
            "type": "Point",
            "coordinates": [e.latlng.lng, e.latlng.lat]
        }
    };
    // Add the new feature to the geojson layer
    geojsonLayer.addData(newFeature);
}
map.on('click', onMapClick);

Nothing fancy here, just a map which adds markers when you click. Now let’s add automerge:

We add a bunch of imports, the goal here will be to sync between tabs of the same browser. Automerge announced an automerge-repo library to help with all the wiring-up, so let’s try it out!

import { DocHandle, isValidAutomergeUrl, Repo } from '@automerge/automerge-repo'
import { BroadcastChannelNetworkAdapter } from '@automerge/automerge-repo-network-broadcastchannel'
import { IndexedDBStorageAdapter } from "@automerge/automerge-repo-storage-indexeddb"
import { v4 as uuidv4 } from 'uuid';

These were just import. Don’t bother too much. The next section does the following:

• Instantiate an “automerge repo”, which helps to send the right messages to the other peers if needed ;
• Add a mechanism to create and initialize a repository if needed,
• or otherwise look for an existing one, based on a hash passed in the URI.

// Add an automerge repository. Sync to 
const repo = new Repo({
    network: [new BroadcastChannelNetworkAdapter()],
    storage: new IndexedDBStorageAdapter(),
});

// Automerge-repo exposes an handle, which is mainly a wrapper around the library internals.
let handle: DocHandle<unknown>
const rootDocUrl = </span><span class="si">${</span><span class="nb">document</span><span class="p">.</span><span class="nx">location</span><span class="p">.</span><span class="nx">hash</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="mf">1</span><span class="p">)</span><span class="si">}</span><span class="sb">
if (isValidAutomergeUrl(rootDocUrl)) {
    handle = repo.find(rootDocUrl);
    let doc = await handle.doc();
    // Once we've found the data in the browser, let's add the features to the geojson layer.
    Object.values(doc.features).forEach(feature => {
        geojsonLayer.addData(feature);
    });
} else {
    handle = repo.create()
    await handle.doc();
    handle.change(doc => doc.features = {});
}

Let’s change the onMapClick function:

function onMapClick(e) {
    const uuid = uuidv4();
    // ... What was there previously
    const newFeature["properties"]["id"] = uuid;

    // Add the new feature to the geojson layer.
    // Here we use the handle to do the change.
    handle.change(doc => { doc.features[uuid] = newFeature});
}

And on the other side of the logic, let’s listen to the changes:

handle.on("change", ({doc, patches}) => {
    // "patches" is a list of all the changes that happened to the tree.
    // Because we're sending JS objects, a lot of patches events are being sent.
    // 
    // Filter to only keep first-level events (we currently don't want to reflect
    // changes down the tree — yet)
    console.log("patches", patches);
    let inserted = patches.filter(({path, action}) => {
        return (path[0] == "features" && path.length == 2 && action == "put")
    });

    inserted.forEach(({path}) => {
        let uuid = path[1];
        let newFeature = doc.features[uuid];
        console.log(Adding a new feature at position </span><span class="si">${</span><span class="nx">uuid</span><span class="si">}</span><span class="sb">)
        geojsonLayer.addData(newFeature);
    });
});

And… It’s working, here is a little video capture of two tabs working together :-)

It’s very rough, but the point was mainly to see how the library can be used, and what the API looks like. I’ve found that :

• The patches object that’s being sent to the handle.on subscribers is very chatty: it contains all the changes, and I have to filter it to get what I want.
• I was expecting the objects to be sent on one go, but it’s creating an operation for each change. For instance, setting a new object to a key will result in multiple events, as it will firstly create the object, and the populate it.
• Here I need to keep track of all the edits, but I’m not sure how that will work out with for instance the offline use-case (or with limited connectivity). That’s what I’m going to find out next week, I guess :-)
• The team behind Automerge is very welcoming, and was prompt to answer me when needed.
• There seem to be another API Automerge.getHistory(), and Automerge.diff() to get a patch between the different docs, which might prove more helpful than getting all the small patches.

We’ll figure that out next week, I guess!

We’ve just released the version 1.9.2 of uMap, that includes a new experimental type of layer: choropleth!

To test it, just select this new type in the dropdown

Then you’ll find new advanced properties to configure it:

Among those properties, the only mandatory is the “property value”, that will tell uMap which layer property to use for computing the choropleth classes.

Optionally, you can define the color palette to use (they come from the color brewer ones) the number of classes you want and the algorithm (from simple-statistics to use for computing the class breaks (which you can also define by hand in the raw input).

It’s quite experimental, so please test it and give feedback!

Other changes in this release include the ability to hide a layer from the caption and a few enhancements for heatmap layers (context).

In the previous episode…

We finally managed to tackle a very popular feature request: datalayers’ fine-grained permissions 🎉. This is a huge step forward, allowing for a given map owner to only open a particular datalayer to edition. It will help people with contributive maps who need to setup a stable/fixed base layer. It also paved the way for even more control over the objects that are allowed for addition and/or edition. Please share with us your desired workflows.

On the UX side of the project, we made a couple of adjustments and fixes to make the editor more intuitive and consistent. Do you see these new crispy icons on the screenshot above? Hopefully it will bring more users, hence more contributors! A couple of new faces jumped in recently and we’re so happy about that 🤗.

You can also look up for icons by name in the ‘Shape properties’ panel, one of our next steps will be to ease icons’ management and additions, another long-awaited feature:

We receive a lot of feedback (and bug reports) from the OSM France forum too which definitely helps us to improve the product in terms of user experience and to prioritize the roadmap. If you are working for the French public sector, do not hesitate to reach out and share your experience.

Stay tuned for the next additions, a brand new API is coming! We’ll be at the NEC - Numérique En Communs event (Bordeaux, France), on October 19th and 20th. See you there for more news and announcements!

Since a few month, uMap has been integrated in a French state incubator, so things are moving quite a lot!

uMap is now ten years old, and is deployed on many instances around the world. The one I know well is hosted by OSM France, and is close to reach one million maps created and 100.000 users.

This incubation program is ported by the French “Accélérateur d’initiatives citoyennes”, it includes coaches and a small budget for non tech needs (UI/UX…). One goal of this program is to find financial support for uMap development and maintainance. A French administration, the Agence pour la cohésion des territoires, is the first uMap financial backer since a few months. This allowed us to put up a small team to work, part time, in uMap:

• two devs: David Larlet and myself
• a UX designer: Aurélie Jallut
• a bizdev, which is yet to hire (interested ? Contact me!)

That’s great news! Until then, uMap was 100% developed on my spare time.

uMap is used a lot by French public agents, and this explains the support from the French state, to make this tool better, and more official. For this, a first step is an “official” instance for public workers:

https://umap.incubateur.anct.gouv.fr/fr/

We’ll be at the NEC - Numérique En Communs event (Bordeaux, France), on October 19th and 20th. See you there for more news and announcements!

What’s new in uMap, then ?

First, a huge cleaning, upgrade and bug fight in uMap code. Since a few years, my time available for uMap has been very low (I’ve been a baker for two years…), so the code urgently needed more love.

What else? Here are a few of the notable changes made recently in uMap, let’s go!

Docker image

Finally!

Custom overlay

Anonymous edit link sent by email

After years of people losing their secret edit link, now there will be no more excuse…

Facet search

Chose some properties, uMap will compute all the values and let people filter data.

Permanent credit

A credit that will display in the bottom left corner:

Starred maps

A cool way to keep tracks of maps made by others!

My Dashboard page

Very basic version of a dashboard page, where to retrieve all our maps, with more metadata and actions than the previous flat maps list.

Better control of default map view

Allow to edit basic user profile information

Useful for changing the username and adding more than one OAuth provider.

Also

• allow to control icon opacity
• allow to sort reverse (by adding a - before the property)
• allow to control links target: same window, new tab, parent window
• add Ctrl-Shift-click shortcut to edit features’s datalayer when clicking on the feature shape
• better natural sort of features
• allow non ascii chars in variables
• Make fromZoom and toZoom options available for all layers
• When map has max bounds set, use those bounds for limiting search

Full changelog.

What’s next ?

A user research session has started by Raywan (who helped us on the bizdev part) and Aurélie, targeted mainly on public workers, but not exclusively.

Some topics on the pipe, from this research and from long waited features:

• concurrent live editing of a map
• define permissions at the datalayer level
• better icon management
• print a map
• UX revamp
• better user documentation
• teams?
• attachements support?

uMap issue management is on Github: https://github.com/umap-project/umap/issues Public roadmap is here: https://github.com/orgs/umap-project/projects/1/views/1

If you want to give your opinion on what should be done first, please add emojis in the issues list. And if something is missing from the list, please create new ones to share your ideas!

How to contribute ?

It’s now finally possible to support uMap by donating on Liberapay or Open Collective. All amounts are welcome!

• https://liberapay.com/uMap/
• https://opencollective.com/uMap

Help translating uMap in your language. uMap is available in 58 languages but many still lack between 15 to 20% to be complete.

https://www.transifex.com/openstreetmap/umap/dashboard/

And of course, contribute to the code!

https://github.com/umap-project/umap/

Cheers, and thanks for your warm support since ten years!