The following high level description will help to basically understand the OSM rendering process. Even if possibly imprecise or outdated in some part, it should allow to rationalize the implemented design.
A map image shown in a browser is built up of many tiles, which are little square images all rendered with a variant of the Mercator projection called Web Mercator, identified as EPSG:3857 or EPSG:900913. This produces a fast approximation to the truer, but heavier elliptical projection.
A general definition of tiled web map (or slippy map in OpenStreetMap terminology) is reported here.
This link provides an overview of how web maps work.
The following diagram from the OSM Component overview represents all main elements of the OpenStreetMap architecture.
The relevant blocks for the rendering process are the ones represented in
openstreetmap-carto includes the following files and folders:
The rendering process takes its data from a PostgreSQL geodatabase with spatial extension implemented through PostGIS (yellow cylinder in the previous drawing). This DB instance holds a constantly updated planet table space in a different format to the database used on the core OSM database server (represented in green in the previous drawing) and is populated by running an osm2pgsql script on minutely diffs. Osm2pgsql acts as ETL, converting OpenStreetMap incremental data to PostGIS-enabled PostgreSQL DB and is able to manage incremental updates of the database as well as to perform an initial load when needed, keeping the PostGIS instance updated or fully refreshing it (in case of periodic database re-import or following a possible major change in openstreetmap-carto that requires reloading the database).
The following diagram represents the process to populate the PostGIS instance with OSM data though osm2pgsql.
|OSM data extract|
OSM data extract is a
.osm file in XML format (JOSM file format) created with JOSM or downloaded from OSM; alternatively it can be a
.pbf file in compressed binary format (Protocolbuffer Binary Format), downloaded from sites like Geofabrik.
openstreetmap-carto.lua is a Lua script invoked by osm2pgsql for data normalization, removal and aggregation. While some standard data management is hardcoded in osm2pgsql, most of the transformations are scripted in openstreetmap-carto.lua, which also covers semantic corrections of features.
openstreetmap-carto.style is a text configuration file of osm2pgsql. It lists all columns which are available in the PostGIS DB tables, to be used by the openstreetmap-carto rendering process. Specifically, any DB field used in project.mml shall match a description in openstreetmap-carto.style. openstreetmap-carto.style is the .style file for OpenStreetMap Carto.
openstreetmap-carto.lua and openstreetmap-carto.style should very rarely be changed. In general, the capability of osm2pgsql to store tags not referred by openstreetmap-carto.style in an hstore column named tags reduces to the minimum the need of adding elements to the openstreetmap-carto.style list, which should be confined to report all very common tags and all columns used for filtering, with all other ones relying on hstore1. Conversely, hstore cannot be used to access objects that do not have at least one key that is in the openstreetmap-carto.style list. For example, if we have an object with
man_made=pipeline (notice that man_made is in the list) and
location=underground (with location not in the list), we would be able to access underground even though location is not in the list. However, we wouldn’t be able to access an object only tagged
emergency=ambulance_station if emergency is not in the list.2
Any modification to openstreetmap-carto.lua or to openstreetmap-carto.style requires a full database re-import (long running batch process which, thanks to the availability of hstore, is a very infrequent operation currently).
The usage of openstreetmap-carto.lua allows to refefine the transformations avioding to rely to the ones hardcoded in osm2pgsql which might be uncomplete or invasive.
Some features in OSM need a specific preprocessing because of their complexity or even to try fixing sparse issues which are currently present in the available data, like unclosed polygons for complex and relevant features or imprecisions of coastlines; also, assembling different parts into a usable whole is needed to simplify rendering. So, features like land polygons, Antarctic ice sheet outlines, world boundaries, country boundaries and offshore land lines are periodically processed offline and converted into shapefiles, which need to be separately rendered instead of managing related data directly into PostgreSQL.
For this, all periodically preprocessed shapefiles data derived from OSM shall be downloaded into a specific folder to be locally accessed by the rendering engine and also indexed for improved search performance.
Shapefiles currently used by OSM for rendering the standard map can be found in OpenStreetMapData, Natural Earth and Planet OSM (notice we are now using https with openstreetmap sites). Additional information available here, here and here.
The process adopted to download and index the needed shapefiles is the following:
|shapefiles||get-shapefiles.py||shapefiles data directory|
The core rendering software currently used by OpenStreetMap is Mapnik, which reads the available data fonts, including the PostGIS database and the shapefiles included in the data directory and then generates the tile raster images (tiles) basing on a proprietary XML stylesheet.
Exploiting a PostGIS database as the backend provides efficient and flexible retrieval from a large amounts of data, allowing optimizations relate to the interaction of PostGIS SQL spatial queries and Mapnik’s layers, rules, and filters.
Produced tiles are then delivered through a custom Apache module named mod_tile, which is responsible for serving tiles and for requesting the rendering of tiles if they aren’t already available in cache or if they have changed since.
Apache provides the front end web server that handles requests from your web browser and passes the request to mod_tile, which in turn checks if the tile has already been created and is ready for use or whether it needs to be updated due to not being in the cache already. If it is already available and does not need to be rendered, then it immediately sends the tile back to the client. If it does need to be rendered, then it will add it to a render request queue, and when it gets to the top of the queue, a tile renderer will render it and send the tile back to the client.
In order to efficiently serve tiles over Internet, OSM exploits more renderers and a CDN (Content Delivery Network) implemented through multiple frontend web caching proxies running Squid and/or TileCache.
OpenStreetMap Carto adopts file formats that are much easier to maintain than the target XML file processed by Mapnik. These files can be directly edited by contributors and a code review can be performed via GitHub.
OpenStreetMap Carto styles are in CartoCSS format, a CSS-like syntax which includes possibly all Mapnik style capabilities and supports Mapnik datasources. CartoCSS can be compiled to the native Mapnik format through a parsing process.
Other than the CartoCSS styles, OpenStreetMap Carto provides a project definition file which contains the core metadata to the project as well as a reference to its sources (vector tiles, shapefiles, PostGIS, etc.) and all the CartoCSS stylesheets it uses. It dscribes the layers and includes the PostGIS queries for each layer; this file is in YAML format and named project.mml.
The conversion of the OpenStreetMap Carto source files into the XML Mapnik file is made by a tool named Carto. The old versions of Carto were able to process a project definition file in JSON format (and not YAML), so a preprocessing of project.mml (YAML format, more readable) to the same name in JSON format was needed and a simple tool named yaml2mml.py did this. The newer versions of Carto are directly capable of processing project.mml in YAML format.
The output of the Carto compiler is the Mapnik XML file, merging the definitions in project.mml together with all referenced styles in .mms files and all shapefile links; the obtained XML file is in final format, to be directly processed by Mapnik.
|osm-carto CartoCSS styles (.mml)||carto||Mapnik XML|
As an example, at the time of writing these are the sizes:
|113189 bytes||project.mml||all *.mms: 234773 bytes|
Mapnik reads the following sources to render the tiles:
The process to generate the Mapnik XML file from the OpenStreetMap Carto sources is the following:
|shapefiles data directory|
Notice that OpenStreetData uses the Web Mercator projection (defined in project.mml and then compiled into the Mapnik XML file). It has the effect to distort the size of objects as the latitude increases from the Equator to the poles, where the scale becomes infinite. Therefore, for example, landmasses such as Greenland and Antarctica appear much larger than they actually are relative to landmasses near the equator, such as Central Africa.
The development environment based on Kosmtik reflects the OSM architecture through a local toolchain.
|project.mml||osm-carto CartoCSS styles (.mss)|
|PostgreSQL PostGIS||Kosmtik||Web images|
|localconfig.json||shapefiles data directory|
Kosmtik includes Carto, node-mapnik and an internal node-js based tileset web service.
Refer to Installing a Docker image of Kosmtik with Ubuntu or Windows for further information on the Kosmtik setup needed for OpenStreetMap Carto.