Welcome to the sixth OpenStreetMap NextGen development diary.

This week, I continued the preparation of the project for the first development release scheduled for the end of this month 🔨.

🔖 You can read my other development diaries here:
@NorthCrab/diary/

📖 This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

🛈 This independent initiative is not affiliated with the OpenStreetMap Foundation.

In Case You Missed It…

OpenStreetMap-NG is planned to have its first development release at the end of this month, May. After this milestone, the project will be open for new contributors! My current work focuses on delivering on that promise, finishing the core functionalities, and stabilizing the code.

Originally posted in diary #5.5.

Finishing Up the Elements Sidebar

The elements sidebar has been mostly finished. The “Part of” and “Members/Nodes” sections are now more consistent in design and have received pagination support. The orange-colored map visualization is also now working.

… See full entry

Welcome to my fifth (and a half) OpenStreetMap NextGen development diary.
Tomorrow, I’m returning home and I’ll be able to resume work at full speed 🔥.

This is a short edition of the development diary.

🐙 This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

Intro

For the past 13 days I have been on a journey of finding a new place to rent. Without my home office, I wasn’t able to become productive. The place I’m staying at doesn’t have a good office spot and being on my laptop doesn’t help. However, I am now very motivated to get back to work and push even harder!

May Will Be Big

At the end of May, OpenStreetMap-NG will include necessary functionality to run on a testing server, as well as to invite new contributors into the project. Starting with 6th of May, I won’t have any time-consuming plans for this month so I’ll do my best to wrap everything up. What’s exactly left has been described in Diary #5 Short-Term Development Plan. I have already started to prepare the All-in-One Contributor Guide which will also be finished up (it currently lacks backend/frontend-specific guides). This is going to be the first major milestone of the project!

Project Sponsors 🏅

I was happily surprised to see new faces even during my lower activity period. I will do everything to deliver the promised results. As always, thank you for supporting the project, monetarily, and with staring the project on GitHub!

Currently, the project is sponsored by 13 people!
Five private and four public donors on Liberapay, and four public on GitHub Sponsors.

• ~1847430, gileri (Eric G.), LaoshuBaby (快乐的老鼠宝宝), RedAuburn (Harry Bond), ~1819511, AurimasF (Aurimas Fišeras), edpichler (Edu), starsep

Disclaimer

This project is not affiliated with the OpenStreetMap Foundation. It’s an independent and community-sponsored initiative.

I am currently on a visit to Ireland 🇮🇪 and a lack of proper office space makes it difficult to stay productive. I will try to prepare something cool to show off this week. Sorry for keeping you waiting!

🍟

Welcome to my fifth OpenStreetMap NextGen development diary.
This week has been mostly focused on GPS Traces 🛰.

🔖 You can read my other development diaries here:
@NorthCrab/diary/

🐙 This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

GPS Traces Demo

It’s best to experience the refreshed traces in a video form, so I prepared a short demo (no audio):
⏯️ https://files.monicz.dev/osm/traces-demo.mp4

For comparison, here’s how the same trace looks on the current website:
@bielebog/traces/11326871

You will quickly notice the super-fast upload speed and the new trace animations. If there’s something wrong with the file you attached, you will receive instant feedback on the upload page. One new feature is possibility to edit trace name. Previously, this feature has been hidden behind API 0.6.

One more planned feature is rendering the map behind the trace animation. Now that the system works on individual coordinates, it will be fairly easy to implement. Traces without a human-understandable point of reference are not as useful as they could be.

Unified Traces URLs

Let’s start with discussing the current URL routes.

• If you want to access a way, you visit /way/<ID>
• If you want to access a note, you visit /note/<ID>
• If you want to access a trace, you visit /user/<USER>/traces/<ID>

Which is not consistent. OpenStreetMap-NG unifies this experience by introducing a new URL route: /trace/<ID>. All existing URLs remain backwards compatible and are automatically redirected.

Short-Term Development Plan

There’s just a few things left before reaching the core feature parity with the existing website. Those are the things I want to finish before inviting new contributors ツ.

• Elements sidebar (50% work in progress)
• User Diaries
• User Profiles
• Applications (OAuth) settings

Contributor Benefits

… See full entry

Welcome to my fourth OpenStreetMap NextGen development diary.
Sorry for being a day late! I wanted to finish up one of the new features which caused this small delay. ✨

🔖 You can read my other development diaries here:
@NorthCrab/diary/

🐙 My work is available publicly on this GitHub repository:
https://github.com/Zaczero/openstreetmap-ng

Let’s summarize the last week’s work:

Client-side Trace Images

While migrating the traces functionality, I came up with an amazing and seemingly obvious idea. Why not make trace images SVGs and render them client-sided? This feature has few significant advantages: even faster trace uploading, no additional disk usage, unlimited customization, infinite resolution, faster page loading. And so here it is:

… See full entry

Welcome to my third OpenStreetMap NextGen development diary.
This week has been super busy and I can’t wait to show you the progress 🧑‍🍳!

You can subscribe to my diary updates on RSS here:
@NorthCrab/diary/rss

Video Presentation 🎉

This week I was focused on integrating features together, as we get closer and closer to the first development release. I am super happy with how things are progressing and I decided to record a short video presentation (3 minutes).

⬇ Click below to play ⬇

or click here: https://peertube.monicz.dev/w/kGnomi7LTveXZNaaQtEwH6

Refreshed Changeset UI

… See full entry

Welcome to my second OpenStreetMap NextGen development diary. I am ready to highlight this week’s progress — and there was a lot 👷!

You can subscribe to my diary updates on RSS here:
@NorthCrab/diary/rss

Finished Settings Page

Last week I showcased initial progress on the new settings page for OpenStreetMap. This week, I’ve finished it. Here’s what’s new. Keep in mind that if you believe some features should be done differently, there will be a period of public testing where I’ll collect this kind of feedback 😉.

Click here to view it in full screen.

Aside from the improved layout, I’ve introduced two notable changes. First, the application language selection has been redesigned; it’s now a simple dropdown selector. I believe the current system (where you input locale names by hand) is difficult for less technical people to understand, and I want to make the website more accessible to everyone!

… See full entry

Today I decided to introduce a new format for sharing OpenStreetMap-NextGen development progress with the community. I’ll post weekly/bi-weekly updates highlighting changes and the current project status. Since this is the first update, I’ll cover some recent highlights.

You can subscribe to my diary updates on RSS: link.

New Settings Page (⭐ Highlight)

I’ve begun migrating the settings/preferences section. My goal is to streamline this experience, as I’ve found the current system a bit complex. Surprisingly, many users don’t know it’s possible to change the default editor — I want to make this more obvious.

A new menu on the left of the screenshot (hidden, not yet finished) will provide clear navigation between general, 2FA, OAuth, and other settings.

This page is still work in progress. I intend to add a help text explaining how to contribute to translations and that the translations are made by the community.

… See full entry

As part of my commitment to OpenStreetMap-NextGen migration, I undertook a comprehensive security review of the existing OpenStreetMap website. I followed the principles of coordinated vulnerability disclosure, working directly with the maintainers to responsibly report my findings. Today, I’m making the details of this process public and verifying the status of the fixes.

Disclosure Timeline

• 2023-11-04 - Contacted Ruby security maintainers & disclosed the timeline publicly
• 2023-11-08 - Maintainers acknowledged the report
• 2023-12-04 - Reported additional vulnerabilities with a 3-month deadline
• 2023-12-06 - Maintainers acknowledged the additional report
• 2024-03-02 - Publicize vulnerability details
• 2024-03-15 - Publicize vulnerability details

---

1. Plain-Text Authentication Token Storage

Authorization tokens (oauth, session, user tokens) are stored in plain text. Read access to the database allows full impersonation of any user.

Status as of 2024-03-15: Partially vulnerable (oauth still depends on plain text storage)

NextGen Codebase Status: Fixed (all authorization tokens and credentials are hashed or encrypted for secure storage)

2. Insecure Email Reply Address Tokenization

The tokens used to ensure the authenticity of email replies are too short (24-bit), making them susceptible to brute-force attacks. An attacker could potentially guess the token and impersonate a legitimate user in email conversations. This vulnerability is especially dangerous if the attacker already has access to a conversation’s metadata (allowing for complete security bypass).

Status as of 2024-03-15: Fixed (now with 48-bit security)

NextGen Codebase Status: Fixed (128 or 256-bit security; undecided)

3. Unbounded GPX Extraction Denial of Service

… See full entry

Today marks a milestone in the development of OpenStreetMap NextGen. After months of rigorous development, I conducted the 1^st OpenStreetMap NextGen performance benchmark, a crucial step towards realizing the vision of a more robust, efficient, and user-friendly OpenStreetMap.

The focus of today’s benchmark was on evaluating static and unauthenticated requests. Since this core functionality is unlikely to change significantly during future development, it’s the perfect time to test it.

Future benchmarks will focus on timing authenticated requests as well as API 0.6.

What was measured

The benchmark analyzed request processing speed, excluding network and client latency. Both osm-ruby and osm-ng support the X-Runtime response header, which tracks how long it takes to process a request and generate a response.

Here’s a general breakdown of a typical static request processing:

… See full entry

I am really excited to finally share some tangible progress on OpenStreetMap NextGen! While I have been working on this project almost day-by-day for the past 3 months, just recently I reached the point of launching the website (for now locally) and begun development of its frontend features.

OpenStreetMap-NG has grown immensely in scope since the original announcement. Now the project not only focuses on core performance, usability, and accessibility but also on improving the user-facing interface and experience. I want this project to truly stand by its name, providing NextGen value to all aspects of the website. And so far, it’s on the right track.

So far, the number of improvements is counted in hundreds, including but not limited to various bug-fixes, performance optimizations, privacy, and security enhancements. And the list keeps on growing!

… See full entry

I have started an independent collection of OSM SLA statistics. Approximately once a month, I will publish my results with the aim of enhancing transparency regarding the reliability of OSM services. I use uptime-kuma to run monitoring. I also verify connectivity with non-OSM services (to prevent false positives). The current configuration includes checking the availability of openstreetmap-website and openstreetmap-cgimap (API). Tile layer availability is not currently included in the checks. The health-check resolution is set to 30 seconds, and the checks are executed from a single server in the Hetzner datacenter in Germany. For the endpoint to be marked unavailable, two consecutive checks must fail. This should be well-representative of an average user experience.

Summary

Total API downtime: 10 minutes and 37 seconds

API 31D SLA: 99.976%

Total website downtime: 30 minutes and 6 seconds

Website 31D SLA: 99.932%

Note that some functionalities of the website require API to also be available.

Details

2024-01-02 11:30:00 - 2024-01-02 11:34:32

• Total downtime: 4 minutes 32 seconds
• 🌐 Website unavailable

2024-01-09 12:51:39 - 2024-01-09 12:53:10

• Total downtime: 1 minute 31 seconds
• 🌐 Website unavailable

2024-01-09 12:56:58 - 2024-01-09 12:59:28

• Total downtime: 2 minutes 30 seconds
• 🌐 Website unavailable

2024-01-09 13:07:18 - 2024-01-09 13:07:48

• Total downtime: 30 seconds
• 🌐 Website unavailable

2024-01-09 13:09:57 - 2024-01-09 13:10:27

• Total downtime: 30 seconds
• 🌐 Website unavailable

2024-01-09 13:16:36 - 2024-01-09 13:19:21

• Total downtime: 2 minutes 45 seconds
• 🌐 Website unavailable

2024-01-14 16:10:19 - 2024-01-14 16:11:04

• Total downtime: 45 seconds
• 🌐 Website unavailable

2024-01-14 16:15:55 - 2024-01-14 16:16:25

• Total downtime: 30 seconds
• 🌐 Website unavailable

2024-01-14 16:17:58 - 2024-01-14 16:18:29

• Total downtime: 31 seconds
• 🌐 Website unavailable

2024-01-14 16:20:02 - 2024-01-14 16:21:02

… See full entry

I have started an independent collection of OSM SLA statistics. Approximately once a month, I will publish my results with the aim of enhancing transparency regarding the reliability of OSM services. I use uptime-kuma to run monitoring. I also verify connectivity with non-OSM services (to prevent false positives). The current configuration includes checking the availability of openstreetmap-website and openstreetmap-cgimap (API). Tile layer availability is not currently included in the checks. The health-check resolution is set to 30 seconds, and the checks are executed from a single server in the Hetzner datacenter in Germany. For the endpoint to be marked unavailable, two consecutive checks must fail. This should be well-representative of an average user experience.

Summary

Total API downtime: 2 hours 34 minutes 13 seconds

API SLA: 99.643%

Total website downtime: 43 minutes 15 seconds

Website SLA: 99.900%

Note that some functionalities of the website require API to also be available.

Details

2023-11-30 10:07:24 - 2023-11-30 10:09:40

• Total downtime: 2 minutes 16 seconds
• 🚩 API unavailable

2023-11-30 10:16:16 - 2023-11-30 10:21:31

• Total downtime: 5 minutes 15 seconds
• 🚩 API unavailable

2023-11-30 21:34:44 - 2023-11-30 21:36:14

• Total downtime: 1 minute 30 seconds
• 🚩 API unavailable

2023-12-01 04:27:18 - 2023-12-01 04:29:00

• Total downtime: 1 minute 42 seconds
• 🚩 API unavailable

2023-12-01 17:26:20 - 2023-12-01 17:29:20

• Total downtime: 3 minutes
• 🚩 API unavailable

2023-12-01 17:45:06 - 2023-12-01 17:50:39

• Total downtime: 5 minutes 33 seconds
• 🚩 API unavailable

2023-12-01 17:50:39 - 2023-12-01 18:04:09

• Total downtime: 13 minutes 30 seconds
• 🚩 API unavailable
• 🌐 Website unavailable

2023-12-01 18:04:09 - 2023-12-01 18:06:51

• Total downtime: 2 minutes 42 seconds
• 🚩 API unavailable

2023-12-01 18:09:38 - 2023-12-01 18:10:23

• Total downtime: 45 seconds
• 🚩 API unavailable

2023-12-01 18:12:25 - 2023-12-01 18:17:09

… See full entry

Over the past few days, my perspective on OpenStreetMap (OSM) has undergone a seismic shift. For the longest time, I held OSM in the highest regard, viewing it as a beacon of transparency and people-centricity amidst a sea of profit-driven tech conglomerates.

Chapter 1: The Banner of Surprises

My view started to waver a couple of days ago when a new donation request banner popped up on the main OSM page. Though its sheer size and prominence were unsettling, the real issue lay in the near-invisible close button. When I voiced concerns on its ready-for-production state, opposition awaited me on the other end. Was this a shift in OSM’s priorities?

Chapter 2: The Amazon Alliance

… See full entry

🗺️🦀 Witajcie, społeczność OpenStreetMap,

Dziś dzielę się z Wami moim najnowszym projektem, osm-yolo-crossings — nowym narzędziem wykorzystującym zaawansowaną technologię AI do samodzielnej detekcji i mapowania przejść dla pieszych (zebry) w OpenStreetMap. Po udanym imporcie budynków w Polsce za pomocą AI, przyszedł czas na poprawę bezpieczeństwa pieszych!

Dzięki mocy detekcji obiektów YOLOv8, to narzędzie automatyzuje mapowanie brakujących przejść dla pieszych na naszych mapach. Z imponującą precyzją wynoszącą ponad 99,7%, jest w stanie zaimportować około 88% wszystkich wykrytych przejść. Pozostałe 12% jest odrzucane z powodu niskiego poziomu pewności. Dzięki inteligentnemu filtrowaniu, system ten jest niesamowicie wydajny. Na przykład, jest w stanie zmapować całą Polskę w ciągu zaledwie dwóch miesięcy, używając pojedynczego serwera bez karty graficznej. To AI pracuje mądrze, a nie ciężko!

Przykładowy zestaw zmian.

… See full entry

🗺️🦀 Hello OpenStreetMap community,

I am excited to share with you my latest invention, osm-yolo-crossings — a new tool that harnesses cutting-edge AI technology to autonomously detect and map pedestrian crossings (zebras) in OpenStreetMap. After the successful AI building import in Poland, it’s now time to expand and improve pedestrian safety!

Leveraging the power of YOLOv8 object detection, this tool is designed to ensure that we no longer miss pedestrian crossings on our maps. With an impressive >99.7% precision rate, it’s able to import around 88% of all detected crossings. The tool discards the remaining 12% due to low confidence levels. Thanks to smart filtering, this system is incredibly efficient. For instance, it can map the entirety of Poland in just about two months using a single server without GPU. This is AI working smart, not hard!

See an example changeset.

… See full entry

🗺️🦀 Witam społeczność OpenStreetMap,

Z zadowoleniem ogłaszam mój najnowszy projekt, osm-budynki-orto-import - w pełni autonomiczne narzędzie do importu budynków, które obecnie funkcjonuje w Polsce. Jest to mój kolejny krok w kierunku uczynienia OpenStreetMap (OSM) bardziej dynamiczną i wydajną platformą.

Narzędzie to zostało zaprojektowane w celu uproszczenia i zwiększenia dokładności procesu importu budynków. System wykorzystuje oficjalne dane budynków w połączeniu ze zdjęciami ortofotograficznymi w celu weryfikacji poprawności danych przed ich zaimportowaniem.

Sercem tego projektu jest zaawansowany model wizji komputerowej, którego precyzja sięga 99,7%. Moim zdaniem dokładność ta przewyższa możliwości większości przeciętnych mapujących, zapewniając szybszy i bardziej niezawodny sposób mapowania struktur.

… See full entry

🗺️🦀 Hello to the OpenStreetMap community,

I am happy to announce my latest project, osm-budynki-orto-import — a fully autonomous building import tool currently in operation in Poland. This is my next step towards making OpenStreetMap (OSM) a more dynamic and efficient platform.

This tool is designed with the objective of making building import process simpler and more accurate. The system utilizes official building data in conjunction with ortophoto imagery to validate the accuracy of the data before importing it.

At the heart of this tool lies an advanced computer vision model, with a precision as high as 99.7%. This accuracy is, in my opinion, superior to the capabilities of most average mappers, providing a faster and more reliable way of mapping structures.

… See full entry

Witaj społeczności OpenStreetMap,

Po publikacji osm-revert, pragnę podzielić się szczegółami na temat mojego nowego projektu, OSM Relatify. Narzędzie to ma na celu uproszczenie procesu edycji relacji transportu publicznego w OpenStreetMap (OSM).

OSM Relatify został zaprojektowany tak, aby mapowanie transportu publicznego było bardziej dostępne i wydajne. Chociaż obecnie skupia się na relacjach autobusowych, wizja OSM Relatify obejmuje szerszy zakres zadań mapowania transportu publicznego.

Jedną z kluczowych cech OSM Relatify jest inteligentna logika wyznaczania tras. Automatycznie tworzy poprawne relacje autobusowe z podanych przystanków i dróg, biorąc pod uwagę takie czynniki jak ulice jednokierunkowe i ronda. To znacznie skraca czas i obniża trudność związaną z zarządzaniem relacjami autobusowymi, czyniąc to narzędzie cennym zasobem zarówno dla nowych, jak i doświadczonych maperów.

… See full entry

Greetings OpenStreetMap community,

Following the release of osm-revert, I’m here to share details about my new project, OSM Relatify. This tool is designed to simplify the process of editing public transport relations within OpenStreetMap (OSM).

OSM Relatify is built with the goal of making public transport mapping more accessible and efficient. While it currently focuses on bus relations, the vision for OSM Relatify is to encompass a broader range of public transport mapping tasks.

One of the key features of OSM Relatify is its smart routing logic. It automatically constructs bus routes from given bus stops and ways, taking into account factors like one-way streets and roundabouts. This significantly reduces the time and complexity involved in managing bus relations, making the tool a valuable asset for both new mappers and seasoned contributors.

… See full entry