Tilemaker PRs: an OpenStreetMap planet to vector tiles in about 3 minutes
Tilemaker, the open-source map generation tool, has achieved a significant performance improvement reducing planet processing time from 14 minutes to 10 minutes with 96 threads.
The open-source map generation tool Tilemaker has received a significant performance optimization that reduces planet processing time by approximately 30%. The improvements, detailed in a recent GitHub pull request, address several bottlenecks in the tile generation process, making it substantially faster while maintaining output quality.
The optimization package combines multiple performance enhancements including faster clipping, simplification and label placement techniques. These changes allow the tool to process the entire planet in just 10 minutes and 55 seconds using 96 threads, compared to the previous time of 14 minutes and 3 seconds.
Thread Management Optimization
One key improvement addresses thread management inefficiencies. Previously, the system would wake up every thread waiting in waitForRoom() after each batch it wrote, even when most of the time the held tiles were still over the limit. At 96 threads, this meant dozens of threads waking, finding no room and sleeping again for every batch.
The fix threads are only woken once the held tiles are back under the limit, reducing context switches from 125 million to 35 million and system time from 7,708 seconds to 2,868 seconds. This optimization alone reduced processing time by 9.5%.
Polygon Clipping Enhancement
The optimization introduces a more efficient polygon clipping approach. Previously, every tile batch that needed a large polygon copied it from the store and clipped it from scratch at every zoom level. The new approach clips polygons with at least 10,000 points once per z8 tile before the tiles are written.
After that the pieces are only read, so there is no lock while tiles are written; a z6 tile's pieces are freed once no tile in it is still to come.
Batch Writing Optimization
The tile writing process was also improved by addressing a bottleneck where the single thread responsible for writing batches in order became the constraint. The solution involves the writer maintaining its own tile data offset instead of calling tellp() for every tile, and giving the stream a 4 MB buffer.
Additionally, OrderedBatches can now take a second step called flushItem, which runs in batch order but is executed by whichever thread is free, preventing it from holding up the first step. This optimization reduced the tile phase time from 323 seconds to 245 seconds.
Memoization for Map Features
A significant performance gain comes from memoization of map features. The system now records, per thread, a tree of what way_function read and which tilemaker functions it called, then replays the calls for ways with identical results without entering Lua.
With this optimization, way_function was replayed for approximately 1.17 billion of 1.22 billion ways, and node_function for about 211 million nodes, resulting in a 21% reduction in wall time when combined with native area calculations.