aselnigu's Diary

LUA - An Interesting Problem When Iterating with `for ... pairs` and the Lua Function `next`

Posted by aselnigu on 7 December 2025 in English. Last updated on 8 December 2025.

Initial Situation

In the forum, a user reports that a road on Crete is not being displayed completely. It quickly becomes clear that the German style is being used for rendering.

Openstreetmap.de operates two tile servers. On both of them, the tiles are faulty. It would be quite a coincidence if this were a hardware issue or a specific data import problem. Therefore, it is very likely that the cause lies within the German style itself.

The only difference between the two road segments is that the visible part contains one additional tag, namely maxspeed:

First Attempt

In the first attempt, I added a tag, namely the surface surface, to the missing segment and forced the German server to re-render the tiles. As a result, the previously missing part of the road appeared. This brought me one step closer, but it is not a solution yet.

Second Attempt

The German style is based on the standard openstreetmap-carto style, which is used on openstreetmap.org for OpenStreetMap maps. It adopts the basic rendering but selectively modifies certain elements to make them more readable for Germans.

I was curious whether the missing road segments were also absent in the original style. Therefore, I rendered them using that style. For this, I used render_single_tile.py and retrieved the necessary information by right-clicking on the corresponding tile at https://tile.openstreetmap.de.

First, I rendered with the German style, where – as expected – the road section was missing:

render_single_tile.py --zxy 17 74577 51762 --stylefile openstreetmap-carto-de/osm-de.xml --outputfile site/rendersinglefile/1.png

Then with the original style, where the rendered tile was also incomplete:

render_single_tile.py --zxy 17 74577 51762 --stylefile openstreetmap-carto/mapnik.xml --outputfile site/rendersinglefile/3.png

From my understanding, this made it clear that the error already occurs when importing the OSM data into the database.

Third Attempt with a Misconception

The data import is performed via osm2pgsql. Here too, the Lua script version in the German style, openstreetmap-carto-flex-l10n.lua, was adapted based on openstreetmap-carto-flex.lua from the standard style.

At the beginning, I quickly noticed that roads with additional tags were displayed. Both LUA files – the original version and the modified German version – have since been extended so that certain keys are filtered out via ignore_keys. I also observed that for all road segments that were not displayed, the tags column in the database was empty. Unfortunately, I focused too much on the tags column. The highway field should also have been filled for osm_id 1340291113 – more on that later.

SELECT osm_id, name, highway, ref, tags
FROM planet_osm_line
WHERE osm_id IN (1340291113, 1340291114);
   osm_id   |                  name                   | highway | ref  |       tags       
------------+-----------------------------------------+---------+------+------------------
 1340291113 | Περάματος - Γαζίου - Perámatos - Gazíou |         | ΕΟ90 | 
 1340291114 | Περάματος - Γαζίου - Perámatos - Gazίou | primary | ΕΟ90 | "maxspeed"=>"40"
(2 rows)

Since I read on switch2osm.org that the original version openstreetmap-carto-flex.lua is not yet actively used, I initially investigated in the wrong direction.

I modified the import script so that whenever no tags were present, "dummy"=>"true" is inserted into the tags column:

local function add_linear(table_name, attrs, geom)
    for sgeom in geom:geometries() do
        attrs.way = sgeom

        if next(attrs.tags) == nil then
            attrs.tags.dummy = "true"
        end

        insert_row(table_name, attrs)
    end
end

In the test database, the result now looks like this. That highway and ref were filled is coincidental, as I discovered later:

SELECT osm_id, name, highway, ref, tags
FROM planet_osm_line
WHERE osm_id IN (1340291113, 1340291114);
   osm_id   |                  name                   | highway | ref  |       tags       
------------+-----------------------------------------+---------+------+------------------
 1340291113 | Περάματος - Γαζίου - Perámatos - Gazíou | primary | ΕΟ90 | "dummy"=>"true"
 1340291114 | Περάματος - Γαζίου - Perámatos - Gazíou | primary | ΕΟ90 | "maxspeed"=>"40"
(2 rows)

After this import, render_single_tile.py renders all road segments correctly, both with the original and the German style.

However, my workaround was considered “bad” – rightfully so, since at that point I still did not understand why the problem occurred in the first place.

Fourth Attempt

At this point, I assumed that the mapnik database queries generating the PNG files for the tiles might have issues if the tags column was empty. I had already mentioned that I had previously overlooked the fact that the highway field was also empty.

To keep it short: I could not find a query that selects only road segments that have tags. However, the highway column seemed to be crucial – and this insight guided me back in the right direction.

Fifth Attempt

The function prepare_columns has been worrying me for some time:

for key, value in pairs(object.tags) do
    if tag_map[key] then
        if (key == 'name') and (L10NLANG ~= nil) then
            attrs[key] = gen_l10n_name(object, islinear, iscountry)
        else
            attrs[key] = value
        end
        found_tag = true
    elseif ignore_type and key == 'type' then -- luacheck: ignore 542
        -- do nothing
    elseif keep_tag(key) then
        attrs.tags[key] = value
        found_tag = true
    end
end

Here we iterate over object.tags while simultaneously modifying object.tags in the l10n daemon during the iteration. I hadn’t questioned this before, since only a single tag is temporarily added. In the actual for loop, the same elements remain present.

The Lua manual states: “You should not assign any value to a non-existent field in a table during its traversal.” But if this were truly problematic, far more roads would be rendered incorrectly.

I examined the Lua source code: lua-5.4.8/src/ltable.c contains the function findindex. The error "invalid key to 'next'" would be raised if an element could not be found while iterating over object.tags. However, in my case, no error is logged.

Sixth Attempt

Since I had no other idea, I finally created a copy of object.tags and used it as the index in the for loop:

local keys = {}
for k in pairs(object.tags) do
    keys[#keys+1] = k
end

According to the Lua Manual, this is actually the recommended approach. Now everything works, even with few tags. At this point, I could basically leave it like this: the error is gone, and the code follows the recommendations of the manual.

However, I would still like to understand why the error only occurs for OSM ways with few tags, while those with many tags run stably.

Seventh Attempt

I’m using my faulty Lua script again and creating two OSM export files for quick testing. One of them contains a problematic way:

sudo -u tile wget -O ways.osm "https://overpass-api.de/api/interpreter?data=[out:xml];way(id:1340291113);(._;>;);out body;"

<osm version="0.6" generator="Overpass API 0.7.62.8 e802775f">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2025-12-06T16:53:40Z"/>

  <node id="313813429" lat="35.3476725" lon="24.8258749"/>
  ...
  <node id="8959531924" lat="35.3475664" lon="24.8266017"/>
  <way id="1340291113">
    <nd ref="313813429"/>
    ...
    <nd ref="8959531917"/>
    <tag k="highway" v="primary"/>
    <tag k="name" v="Περάματος - Γαζίου"/>
    <tag k="ref" v="ΕΟ90"/>
    <tag k="source:ref" v="Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)"/>
  </way>

</osm>

And an adjacent segment with a way that is not problematic:

sudo -u tile wget -O ways.osm "https://overpass-api.de/api/interpreter?data=[out:xml];way(id:1340291114);(._;>;);out body;"

<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2025-12-06T16:53:40Z"/>

  <node id="313813460" lat="35.3465775" lon="24.8360096"/>
  ...
  <node id="8959531917" lat="35.3465228" lon="24.8358648"/>
  <way id="1340291114">
    <nd ref="8959531917"/>
    ...
    <nd ref="4810699410"/>
    <tag k="highway" v="primary"/>
    <tag k="maxspeed" v="40"/>
    <tag k="name" v="Περάματος - Γαζίου"/>
    <tag k="ref" v="ΕΟ90"/>
    <tag k="source:maxspeed" v="sign"/>
    <tag k="source:ref" v="Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)"/>
  </way>

</osm>

Now I add print() statements to the Lua script. At the beginning of the loop, I print the tag that is currently being processed. Additionally, I print all object.tags together before and after the call to gen_l10n_name, where object.tags is modified.

for key, value in pairs(object.tags) do
    print(string.format("key: %s, value: %s", tostring(key), tostring(value)))

    if tag_map[key] then
        if key == 'name' and L10NLANG ~= nil then

            print("  --> Processing 'name' key with L10NLANG")

            for akey, avalue in pairs(object.tags) do
                print(string.format("    [before] akey: %s, avalue: %s", tostring(akey), tostring(avalue)))
            end

            attrs[key] = gen_l10n_name(object, islinear, iscountry)

            for zkey, zvalue in pairs(object.tags) do
                print(string.format("    [after] zkey: %s, zvalue: %s", tostring(zkey), tostring(zvalue)))
            end
         
            else
              attrs[key] = value
            end
            found_tag = true
        elseif ignore_type and key == 'type' then -- luacheck: ignore 542
            -- do nothing
        elseif keep_tag(key) then
            attrs.tags[key] = value
            found_tag = true
        end
    end

    if not found_tag then
        return nil
    end

    return attrs
end

Using

osm2pgsql --create -d gis --slim --output flex -S /srv/tile/openstreetmap-carto-de/openstreetmap-carto-flex-l10n.lua way113.osm

and

osm2pgsql --create -d gis --slim --output flex -S /srv/tile/openstreetmap-carto-de/openstreetmap-carto-flex-l10n.lua way114.osm

I can now test quite quickly.

I noticed that for a way with six tags (as in osm_id 1340291114), the order of the tags remains consistent during the loop iteration.

key: ref, value: ΕΟ90
key: highway, value: primary
key: maxspeed, value: 40
key: source:ref, value: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
key: name, value: Περάματος - Γαζίου
  --> Processing 'name' key with L10NLANG
    [before] akey: ref, avalue: ΕΟ90
    [before] akey: highway, avalue: primary
    [before] akey: maxspeed, avalue: 40
    [before] akey: source:ref, avalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [before] akey: name, avalue: Περάματος - Γαζίου
    [before] akey: source:maxspeed, avalue: sign
    [after] zkey: ref, zvalue: ΕΟ90
    [after] zkey: highway, zvalue: primary
    [after] zkey: maxspeed, zvalue: 40
    [after] zkey: source:ref, zvalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [after] zkey: name, zvalue: Περάματος - Γαζίου
    [after] zkey: source:maxspeed, zvalue: sign
key: source:maxspeed, value: sign

On a second run, the order can be different. This is also stated in the Lua Manual: “The order in which the indices are enumerated is not specified, even for numeric indices.”

However, during a single loop iteration, the order of object.tags always remains the same – as shown in my print() outputs for key, akey, and zkey.

key: source:ref, value: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
key: name, value: Περάματος - Γαζίου
  --> Processing 'name' key with L10NLANG
    [before] akey: source:ref, avalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [before] akey: name, avalue: Περάματος - Γαζίου
    [before] akey: maxspeed, avalue: 40
    [before] akey: source:maxspeed, avalue: sign
    [before] akey: ref, avalue: ΕΟ90
    [before] akey: highway, avalue: primary
    [after] zkey: ref, zvalue: ΕΟ90
    [after] zkey: source:ref, zvalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [after] zkey: name, zvalue: Περάματος - Γαζίου
    [after] zkey: maxspeed, zvalue: 40
    [after] zkey: source:maxspeed, zvalue: sign
    [after] zkey: highway, zvalue: primary
key: maxspeed, value: 40
key: source:maxspeed, value: sign
key: highway, value: primary

It is quite different with four tags (as in osm_id 1340291113). Here, zkey often changes. When my highway was missing, the situation apparently was like in the following log: Everything started in the order ref, source:ref, name, and highway. Then, when object.tags was modified during the iteration of the name tag, the order changed to source:ref, highway, name, and ref. As a result, highway was not processed in this loop iteration, while ref was processed twice:

key: ref, value: ΕΟ90
key: source:ref, value: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
key: name, value: Περάματος - Γαζίου
  --> Processing 'name' key with L10NLANG
    [before] akey: ref, avalue: ΕΟ90
    [before] akey: source:ref, avalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [before] akey: name, avalue: Περάματος - Γαζίου
    [before] akey: highway, avalue: primary
    [after] zkey: source:ref, zvalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [after] zkey: highway, zvalue: primary
    [after] zkey: name, zvalue: Περάματος - Γαζίου
    [after] zkey: ref, zvalue: ΕΟ90
key: ref, value: ΕΟ90

In the next iteration, highway was processed twice.

key: highway, value: primary
key: source:ref, value: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
key: ref, value: ΕΟ90
key: name, value: Περάματος - Γαζίου
  --> Processing 'name' key with L10NLANG
    [before] akey: highway, avalue: primary
    [before] akey: source:ref, avalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [before] akey: ref, avalue: ΕΟ90
    [before] akey: name, avalue: Περάματος - Γαζίου
    [after] zkey: name, zvalue: Περάματος - Γαζίου
    [after] zkey: ref, zvalue: ΕΟ90
    [after] zkey: source:ref, zvalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [after] zkey: highway, zvalue: primary
key: ref, value: ΕΟ90
key: source:ref, value: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
key: highway, value: primary

After that, highway was not processed again:

key: source:ref, value: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
key: name, value: Περάματος - Γαζίου
  --> Processing 'name' key with L10NLANG
    [before] akey: source:ref, avalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [before] akey: name, avalue: Περάματος - Γαζίου
    [before] akey: highway, avalue: primary
    [before] akey: ref, avalue: ΕΟ90
    [after] zkey: highway, zvalue: primary
    [after] zkey: source:ref, zvalue: Ministerial Decision Γ25871/1963 (ΦΕΚ Β 319/23.07.1963)
    [after] zkey: ref, zvalue: ΕΟ90
    [after] zkey: name, zvalue: Περάματος - Γαζίου

With four tags, in my tests it’s a bit of a gamble whether all tags are processed, whereas with six tags everything is stable.

How can this be explained?

Lua tables consist of an array part for positive integer keys and a hash part for all other keys. Our tags, as strings, fall into the hash part. According to my research, the relevant code for adding elements in lua-5.4.8/src/ltable.c looks like this:

if (f == NULL) {  /* cannot find a free place? */
  rehash(L, t, key);  /* grow table */
  luaH_set(L, t, key, value);
  return;
}

This means: if no free hash slot is found, a rehash() is triggered immediately, potentially redistributing the elements.

Why does this happen more often in some tables?

Lua always chooses the hash size as a power of two (2ⁿ):

lsize = luaO_ceillog2(size);
size = twoto(lsize);

Example: 4 elements

ceil(log2(4)) = 2
2^2 = 4

Elements	Hash size	Free slots
4	4	0

Inserting another element triggers a rehash() and may rearrange the elements.

Example: 6 elements

ceil(log2(6)) ≈ ceil(2.58) = 3
2^3 = 8

Elements	Hash size	Free slots
6	8	2

Inserting or deleting elements here does not trigger a rehash(). The order of elements remains stable.

Example: 8 elements

ceil(log2(8)) = 3
2^3 = 8

Elements	Hash size	Free slots
8	8	0

Inserting another element triggers a rehash() again and may rearrange the elements. However, since the table is now larger, the likelihood of a significant portion of elements being “lost” or imported incorrectly is lower.