Description of the OSM rendering process

OSM architecture

The following high level description will help to basically understand the OSM rendering process. Even if possibly imprecise or outdated in some part, it should allow to rationalize the implemented design.

A map image shown in a browser is built up of many tiles, which are little square images all rendered with a variant of the Mercator projection called Web Mercator, identified as EPSG:3857 or EPSG:900913. This produces a fast approximation to the truer, but heavier elliptical projection.

A general definition of tiled web map (or slippy map in OpenStreetMap terminology) is reported here.

This link provides an overview of how web maps work.

The following diagram from the OSM Component overview represents all main elements of the OpenStreetMap architecture.

The relevant blocks for the rendering process are the ones represented in yellow.

Structure of openstreetmap-carto

openstreetmap-carto includes the following files and folders:

• one project/layer description file in yaml format (project.mml)
• the same project/layer description file converted into the json format (project.mml)
• one or more CartoCSS stylesheets (e.g., style.mss and all the others)
• a symbols subdirectory for storing svg and png files
• a scripts subdirectory including all scripts
• openstreetmap-carto.style/openstreetmap-carto.lua: osm2pgsql configuration files
• description files:
  • README.md
  • preview.png
  • LICENSE.txt
  • CARTOGRAPHY.md
  • CONTRIBUTING.md
  • RELEASES.md
  • CODE_OF_CONDUCT.md
  • CHANGELOG.md
  • INSTALL.md
  • DOCKER.md
  • USECASES.md
• support files for the scripts: indexes.yml/road-colors.yaml/indexes.sql/.travis.yml

Description of the rendering process

The rendering process takes its data from a PostgreSQL geodatabase with spatial extension implemented through PostGIS (yellow cylinder in the previous drawing). This DB instance holds a constantly updated planet table space in a different format to the database used on the core OSM database server (represented in green in the previous drawing) and is populated by running an osm2pgsql script on minutely diffs. Osm2pgsql acts as ETL, converting OpenStreetMap incremental data to PostGIS-enabled PostgreSQL DB and is able to manage incremental updates of the database as well as to perform an initial load when needed, keeping the PostGIS instance updated or fully refreshing it (in case of periodic database re-import or following a possible major change in openstreetmap-carto that requires reloading the database).

osm2pgsql supports Lua scripts to perform complex queries and rewrite/unify tags before data enter the database. Lua pre-processing saves further Mapnik processing.

Populating the PostGIS instance

The following diagram represents the process to populate the PostGIS instance with OSM data though osm2pgsql.

		OSM data extract
openstreetmap-carto.style		osm2pgsql		PostgreSQL PostGIS
		openstreetmap-carto.lua

OSM data extract is a .osm file in XML format (JOSM file format) created with JOSM or downloaded from OSM; alternatively it can be a .pbf file in compressed binary format (Protocolbuffer Binary Format), downloaded from sites like Geofabrik.

openstreetmap-carto.lua is a Lua script invoked by osm2pgsql for data normalization, removal and aggregation. While some standard data management is hardcoded in osm2pgsql, most of the transformations are scripted in openstreetmap-carto.lua, which also covers semantic corrections of features.

openstreetmap-carto.style is a text configuration file of osm2pgsql. It lists all columns which are available in the PostGIS DB tables, to be used by the openstreetmap-carto rendering process. Specifically, any DB field used in project.mml shall match a description in openstreetmap-carto.style. openstreetmap-carto.style is the .style file for OpenStreetMap Carto.

openstreetmap-carto.lua and openstreetmap-carto.style should very rarely be changed. In general, the capability of osm2pgsql to store tags not referred by openstreetmap-carto.style in an hstore column named tags reduces to the minimum the need of adding elements to the openstreetmap-carto.style list, which should be confined to report all very common tags and all columns used for filtering, with all other ones relying on hstore¹. Conversely, hstore cannot be used to access objects that do not have at least one key that is in the openstreetmap-carto.style list. For example, if we have an object with man_made=pipeline (notice that man_made is in the list) and location=underground (with location not in the list), we would be able to access underground even though location is not in the list. However, we wouldn’t be able to access an object only tagged emergency=ambulance_station if emergency is not in the list.²

Any modification to openstreetmap-carto.lua or to openstreetmap-carto.style requires a full database re-import (long running batch process which, thanks to the availability of hstore, is a very infrequent operation currently).

The usage of openstreetmap-carto.lua allows to refefine the transformations avioding to rely to the ones hardcoded in osm2pgsql which might be uncomplete or invasive.

Obtaining an indexed image of the shapefiles

Some features in OSM need a specific preprocessing because of their complexity or even to try fixing sparse issues which are currently present in the available data, like unclosed polygons for complex and relevant features or imprecisions of coastlines; also, assembling different parts into a usable whole is needed to simplify rendering. So, features like land polygons, Antarctic ice sheet outlines, world boundaries, country boundaries and offshore land lines are periodically processed offline and converted into shapefiles, which need to be separately rendered instead of managing related data directly into PostgreSQL.

For this, all periodically preprocessed shapefiles data derived from OSM shall be downloaded into a specific folder to be locally accessed by the rendering engine and also indexed for improved search performance.

Shapefiles currently used by OSM for rendering the standard map can be found in OpenStreetMapData, Natural Earth and Planet OSM (notice we are now using https with openstreetmap sites). Additional information available here, here and here.

The process adopted to download and index the needed shapefiles is the following:

shapefiles

get-shapefiles.py

shapefiles data directory

Mapnik rendering

The core rendering software currently used by OpenStreetMap is Mapnik, which reads the available data fonts, including the PostGIS database and the shapefiles included in the data directory and then generates the tile raster images (tiles) basing on a proprietary XML stylesheet.

Exploiting a PostGIS database as the backend provides efficient and flexible retrieval from a large amounts of data, allowing optimizations relate to the interaction of PostGIS SQL spatial queries and Mapnik’s layers, rules, and filters.

Produced tiles are then delivered through a custom Apache module named mod_tile, which is responsible for serving tiles and for requesting the rendering of tiles if they aren’t already available in cache or if they have changed since.

Apache provides the front end web server that handles requests from your web browser and passes the request to mod_tile, which in turn checks if the tile has already been created and is ready for use or whether it needs to be updated due to not being in the cache already. If it is already available and does not need to be rendered, then it immediately sends the tile back to the client. If it does need to be rendered, then it will add it to a render request queue, and when it gets to the top of the queue, a tile renderer will render it and send the tile back to the client.

In order to efficiently serve tiles over Internet, OSM exploits more renderers and a CDN (Content Delivery Network) implemented through multiple frontend web caching proxies running Squid and/or TileCache.

The web interface for browsing the rendered OpenStreetMap data is named Slippy Map. The slippy map is an Ajax JavaScript component running in the browser, which dynamically requests maps from the tile server in the background (without reloading the whole HTML page) to give a smooth slippy zoomy map browsing experience.

Process to convert the project/layer description file

OpenStreetMap Carto adopts file formats that are much easier to maintain than the target XML file processed by Mapnik. These files can be directly edited by contributors and a code review can be performed via GitHub.

OpenStreetMap Carto styles are in CartoCSS format, a CSS-like syntax which includes possibly all Mapnik style capabilities and supports Mapnik datasources. CartoCSS can be compiled to the native Mapnik format through a parsing process.

Other than the CartoCSS styles, OpenStreetMap Carto provides a project definition file which contains the core metadata to the project as well as a reference to its sources (vector tiles, shapefiles, PostGIS, etc.) and all the CartoCSS stylesheets it uses. It dscribes the layers and includes the PostGIS queries for each layer; this file is in YAML format and named project.mml.

The conversion of the OpenStreetMap Carto source files into the XML Mapnik file is made by a tool named Carto. The old versions of Carto were able to process a project definition file in JSON format (and not YAML), so a preprocessing of project.mml (YAML format, more readable) to the same name in JSON format was needed and a simple tool named yaml2mml.py did this. The newer versions of Carto are directly capable of processing project.mml in YAML format.

The output of the Carto compiler is the Mapnik XML file, merging the definitions in project.mml together with all referenced styles in .mms files and all shapefile links; the obtained XML file is in final format, to be directly processed by Mapnik.

		project.mml
osm-carto CartoCSS styles (.mml)		carto		Mapnik XML

As an example, at the time of writing these are the sizes:

113189 bytes	project.mml		all *.mms: 234773 bytes
2077776 bytes	style.xml

Process to render data

Mapnik reads the following sources to render the tiles:

• Mapnik XML, generated by compiling the OpenStreetMap Carto files and including all rendering definitions
• PostGIS data, via queries and configuration included in Mapnik XML
• Shapefiles data, via configuration included in Mapnik XML

The process to generate the Mapnik XML file from the OpenStreetMap Carto sources is the following:

		Mapnik XML
PostgreSQL PostGIS		Mapnik		images
		shapefiles data directory

Some description of the rendering with the standard tile layer is described here and here.

Notice that OpenStreetData uses the Web Mercator projection (defined in project.mml and then compiled into the Mapnik XML file). It has the effect to distort the size of objects as the latitude increases from the Equator to the poles, where the scale becomes infinite. Therefore, for example, landmasses such as Greenland and Antarctica appear much larger than they actually are relative to landmasses near the equator, such as Central Africa.

Development and testing environment

The best platform to perform CartoCSS customizations is Kosmtik. (In the past, TileMill was used.)

The development environment based on Kosmtik reflects the OSM architecture through a local toolchain.

project.mml		osm-carto CartoCSS styles (.mss)
PostgreSQL PostGIS		Kosmtik		Web images
localconfig.json		shapefiles data directory

Kosmtik includes Carto, node-mapnik and an internal node-js based tileset web service.

Refer to Installing a Docker image of Kosmtik with Ubuntu or Windows for further information on the Kosmtik setup needed for OpenStreetMap Carto.