OpenStreetMap Carto Tutorials - OpenStreetMap Carto documentation     Site map    
Installing Kosmtik and OpenStreetMap-Carto on Ubuntu

Installing Kosmtik and OpenStreetMap-Carto on Ubuntu

Introduction

Kosmtik is an open source modular mapping framework for managing CartoCSS rendering of OpenStreetMap stylesheets and creating Mapnik ready raster tile maps. It consists of a JavaScript node module and needs a list of prerequisite software like PostgreSQL, PostGIS, Python, osm2pgsql and Node.js itself. Kosmtik also includes node versions of further software like Mapnik and Carto. It is written by Yohan Boniface and other contributors.

The recommended method to install Kosmtik is via the Docker image and related scripts provided with OpenStreetMap Carto. Alternatively to the Docker installation method, the following procedure can be used to install a Kosmtik development environment of openstreetmap-carto by performing one by one all the required steps on an Ubuntu PC.

Notice that Kosmtik does not currently install on Windows (setup fails on node-mapnik). It can be installed by now on:

  • a Unix server (e.g., Ubuntu)
  • a local UNIX virtual machine
  • a cloud based virtual machine

A PostGIS database is needed and can be installed locally (suggested) or remotely (might be slow, depending on the network).

The goal for this procedure is to use Ubuntu packages and official PPAs whenever possible.

We consider using Ubuntu 20.04.2 LTS Focal Fossa, or 18.04 LTS Bionic Beaver, suggested operating system version.

Other tested O.S. include Ubuntu 16.04 LTS Xenial Xerus, Ubuntu 15.4 Vivid Vervet or Ubuntu 14.04.3 LTS Trusty Tahr (other versions should work). All should be 64-bit computing architecture. Other distributions like Debian might be checked, but could require changes to the installation procedure.

This procedure is updated to the version of OpenStreetMap Carto available at the time of writing. To get the correct installation procedure, the INSTALL history should be checked, considering that the OpenStreetMap Carto maintainers use to keep the INSTALL page updated. Check also the README changelog.

General setup for Ubuntu

Install Ubuntu.

This procedure also supports WSL - Windows Subsystem for Linux. This means that a Windows 10 64-bit PC can be used to perform the installation, after setting-up WSL.

Update Ubuntu

Make sure your Ubuntu system is fully up-to-date:

lsb_release -a

Previous command returns the Ubuntu version.

To update the system:

sudo apt-get update
sudo apt-get -y upgrade

If on a brand new system you also want to do sudo apt-get dist-upgrade && sudo shutdown -r now.

Install essential tools

sudo apt-get -y install ca-certificates gnupg curl unzip gdal-bin \
tar wget bzip2 build-essential clang python3-psycopg2 python3-yaml \
python3-requests postgresql-client

Optional elements:

sudo apt-get -y install munin-node munin protobuf-c-compiler libtiff5-dev
libcairomm-1.0-dev libagg-dev lua5.1 liblua5.1-0-dev

Check prerequisites suggested by openstreetmap-carto.

For the subsequent installation steps, we suppose that cd defaults to your home directory.

Configure a swap

Importing and managing map data takes a lot of RAM and a swap is generally needed.

To check whether a swap partition is already configured on your system, use one of the following two commands:

  • Reports the swap usage summary (no output means missing swap):

    swapon -s
    
  • Display amount of free and used memory in the system (check the line specifying Swap):

    free -h
    

If you do not have an active swap partition, especially if your physical memory is small, you should add a swap file. First we use fallocate command to create a file. For example, create a file named swapfile with 2G capacity in root file system:

sudo fallocate -l 2G /swapfile

Then make sure only root can read and write to it.

sudo chmod 600 /swapfile

Format it to swap:

sudo mkswap /swapfile

Enable the swap file

sudo swapon /swapfile

The Operating System tuning adopted by the OpenStreetMap tile servers can be found in the related Chef configuration.

Check usage of English locale

Run locale to list what locales are currently defined for the current user account:

locale

To set the en_GB locale:

export LANGUAGE=en_GB.UTF-8
export LANG=en_GB.UTF-8
export LC_ALL=en_GB.UTF-8

The exported variables can be put to the file /etc/environment.

New locales can also be generated by issuing:

sudo locale-gen en_GB en_GB.UTF-8
sudo dpkg-reconfigure locales

The following subfolders will be created:

  • kosmtik
  • openstreetmap-carto

Install Git

Git might come already preinstalled sometimes.

git --version # to verify whether git is already installed
sudo apt-get install -y git

Install Node.js

Install Node.js with Ubuntu 20.04 LTS:

# Check installations:
node -v
nodejs -v
npm -v

# If components needs to be installed, run the following:
sudo apt install -y nodejs npm

Go to Check Node.js versions.

Additional notes on Node.js: other modes to install it:

A list of useful commands to manage Node.js is available at a specific page.

The above reported Node.js version also supports installing TileMill and Carto.

Distro version from the APT package manager

The recent versions of Ubuntu come with Node.js (nodejs package) and npm (npm package) in the default repositories. Depending on which Ubuntu version you’re running, those packages may contain outdated releases; the one coming with Ubuntu 16.04 will not be the latest, but it should be stable and sufficient to run Kosmtik and Carto. TileMill instead needs nodejs-legacy (or an old version of node installed via a Node.js version management tool).

For Kosmtik we will install nodejs:

sudo apt-get install -y nodejs npm
node -v 2>/dev/null || sudo ln -fs /usr/bin/nodejs /usr/local/bin/node
nodejs -v
node -v
npm -v

Install Node.js through a version management tool

Alternatively, a suggested approach is using a Node.js version management tool, which simplifies the interactive management of different Node.js versions and allows performing the upgrade to the latest one. We will use n.

Install n:

mkdir -p ~/src ; cd ~/src
git clone https://github.com/tj/n.git
cd n
sudo make install # To uninstall: sudo make uninstall
cd ..

Some programs (like Kosmtik and carto) accept the latest LTS node version (sudo n lts), other ones (like Tilemill) run with v6.14.1 (sudo n 6.14.1).

For Kosmtik we will install the latest LTS one:

sudo n lts

Check Node.js versions

To get the installed version numbers:

node -v
npm -v

Install Kosmtik

Note: with Ubuntu 20.04 LTS, go to the Kosmtik installation from its GitHub repository.

sudo npm install --unsafe-perm -g kosmtik

Go to next installation steps.

Install Kosmtik from its GitHub repository

As alternative installation method, install Kosmtik from its GitHub repository:

sudo npm install --unsafe-perm -g git+https://github.com/kosmtik/kosmtik

In case the installation fails, this is possibly due to some incompatibility with npm/Node.js; to fix this, try downgrading the Node.js version.

Read Kosmtik Install or Update for further information.

Python installation

Check that Python is installed:

sudo apt-get install -y python3 python3-distutils

# Verify Python installation:
python -V
python3 -V

Install Yaml and Package Manager for Python

This is necessary in order to run OpenStreetMap-Carto scripts/indexes.

sudo apt-get install -y python-yaml

pip -V # to verify whether pip is already installed
sudo apt-get install -y python3-pip
python3 -m pip install --upgrade pip

Install Mapnik Utilities

The Mapnik Utilities package includes shapeindex.

sudo apt-get install -y mapnik-utils

Install openstreetmap-carto

mkdir -p ~/src
cd ~/src
git clone https://github.com/gravitystorm/openstreetmap-carto.git
cd openstreetmap-carto

Read installation notes for further information.

Install the fonts needed by openstreetmap-carto

Currently Noto fonts are used.

To install them (except Noto Emoji Regular and Noto Sans Arabic UI Regular/Bold):

sudo apt-get install -y fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted fonts-hanazono ttf-unifont

Installation of Noto fonts (hinted ones should be used if available1):

cd ~/src
git clone https://github.com/googlefonts/noto-emoji.git
git clone https://github.com/googlefonts/noto-fonts.git

sudo cp noto-emoji/fonts/NotoColorEmoji.ttf /usr/share/fonts/truetype/noto
sudo cp noto-emoji/fonts/NotoEmoji-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansArabicUI/NotoSansArabicUI-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoNaskhArabicUI/NotoNaskhArabicUI-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansArabicUI/NotoSansArabicUI-Bold.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoNaskhArabicUI/NotoNaskhArabicUI-Bold.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansAdlam/NotoSansAdlam-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansAdlamUnjoined/NotoSansAdlamUnjoined-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansChakma/NotoSansChakma-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansOsage/NotoSansOsage-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansSinhalaUI/NotoSansSinhalaUI-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansArabicUI/NotoSansArabicUI-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansCherokee/NotoSansCherokee-Bold.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansSinhalaUI/NotoSansSinhalaUI-Bold.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansSymbols/NotoSansSymbols-Bold.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansArabicUI/NotoSansArabicUI-Bold.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/unhinted/ttf/NotoSansSymbols2/NotoSansSymbols2-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/hinted/ttf/NotoSansBalinese/NotoSansBalinese-Regular.ttf /usr/share/fonts/truetype/noto
sudo cp noto-fonts/archive/hinted/NotoSansSyriac/NotoSansSyriac-Regular.ttf /usr/share/fonts/truetype/noto

mkdir NotoSansSyriacEastern-unhinted
cd NotoSansSyriacEastern-unhinted
wget https://noto-website-2.storage.googleapis.com/pkgs/NotoSansSyriacEastern-unhinted.zip
unzip NotoSansSyriacEastern-unhinted.zip
sudo cp NotoSansSyriacEastern-Regular.ttf /usr/share/fonts/truetype/noto
cd ..

sudo apt install fontconfig

At the end:

sudo fc-cache -fv
fc-list
fc-list | grep Emoji

DejaVu Sans is used as an optional fallback font for systems without Noto Sans. If all the Noto fonts are installed, it should never be used.

sudo apt-get install -y fonts-dejavu-core

Read font notes for further information.

Old unifont Medium font

The unifont Medium font (lowercase label), which was included in past OS versions, now is no more available and substituted by Unifont Medium (uppercase). Warnings related to the unavailability of unifont Medium are not relevant2 and are due to the old decision of OpenStreetMap maintainers to support both the past Ubuntu 12.04 font and the newer version (uppercase).

One way to avoid the warning is removing the reference to “unifont Medium” in openstreetmap-carto/style.xml.

Another alternative way to remove the lowercase unifont Medium warning is installing the old “unifont Medium” font (used by Ubuntu 12.10):

mkdir -p ~/src ; cd ~/src
mkdir OldUnifont
cd OldUnifont
wget http://http.debian.net/debian/pool/main/u/unifont/unifont_5.1.20080914.orig.tar.gz
tar xvfz unifont_5.1.20080914.orig.tar.gz unifont-5.1.20080914/font/precompiled/unifont.ttf
sudo cp unifont-5.1.20080914/font/precompiled/unifont.ttf /usr/share/fonts/truetype/unifont/OldUnifont.ttf
sudo fc-cache -fv
fc-list | grep -i unifont # both uppercase and lowercase fonts will be listed

Notice that above installation operation is useless, just removes the warning.

Create localconfig.json

Using your favourite editor, create a file named localconfig.json in the openstreetmap-carto directory, including the following:

cd ~/src
cd openstreetmap-carto
cat > localconfig.json <<\eof
[
    {
        "where": "center",
        "then": [9.111, 45.111, 15]
    },
    {
        "where": "Layer",
        "if": {
            "Datasource.type": "postgis"
        },
        "then": {
            "Datasource.dbname": "gis"
        }
    }
]
eof

If accessing a remote server, also set Datasource.password, Datasource.user and Datasource.host:

cd ~/src
cd openstreetmap-carto
cat > localconfig.json <<\eof
[
    {
        "where": "center",
        "then": [9.111, 45.111, 15]
    },
    {
        "where": "Layer",
        "if": {
            "Datasource.type": "postgis"
        },
        "then": {
            "Datasource.dbname": "gis",
            "Datasource.password": "",
            "Datasource.user": "",
            "Datasource.host": "localhost"
        }
    }
]
eof

Notice the datasource parameters, including user and password, that you might need to change.

Replace coordinates and zoom with your preferred ones within the following line:

        "then": [9.111, 45.111, 15]

For instance, for Liechtenstein you would use:

        "then": [9.5634, 47.1237, 15]

In this example, the default center is (9.111, 45.111) and the default zoom is 15. You can configure the other parameters (like the db ones: dbname, password, user, host). Replace localhost with a remote host name if the PostGIS instance is running remotely. Read local config for further information.

Configure the firewall

If you are preparing a remote virtual machine, configure the firewall to allow remote access to the local port 6789.

If you run a cloud based VM, also the VM itself shall be set to open this port.

Install PostgreSQL and PostGIS

PostgreSQL is a relational database, and PostGIS is its spatial extender, which allows you to store geographic objects like map data in it; it serves a similar function to ESRI’s SDE or Oracle’s Spatial extension. PostgreSQL + PostGIS are used for a wide variety of features such as rendering maps, geocoding, and analysis.

Currently the tested versions for OpenstreetMap Carto are PostgreSQL 10 and PostGIS 2.4:

Also older or newer PostgreSQL version should be suitable.

On Ubuntu there are pre-packaged versions of both postgis and postgresql, so these can simply be installed via the Ubuntu package manager.

sudo apt-get update
sudo apt-get install -y postgresql postgis

Optional components:

sudo apt-get install -y postgresql-contrib postgresql-12-postgis-3 postgresql-12-postgis-3-scripts

You need to start the db:

sudo service postgresql start

Note: used PostgreSQL port is 5432 (default).

Create the PostGIS instance

Now you need to create a PostGIS database. The defaults of various programs including openstreetmap-carto (ref. project.mml) assume the database is called gis. You need to create a PostgreSQL database and set up a PostGIS extension on it.

The character encoding scheme to be used in the database is UTF8 and the adopted collation is en_GB.utf8. (The U&"..." escaped Unicode syntax used in project.mml should work only when the server encoding is UTF8. This is also in line with what reported in the PostgreSQL Chef configuration code.)

sudo -u postgres createuser -s $USER
createdb gis --encoding="UTF8" --lc-collate="en_GB.UTF-8" --lc-ctype="en_GB.UTF-8" --template=template0
psql -d gis -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'

Note: ERROR: invalid locale name: "en_GB.UTF-8" means that en_GB.UTF-8 locale has not been installed. After installing locale, the database shall be restarted in order to be able to load the locale.

Go to the next step.

If in different host:

Set the environment variables

export PGHOST=localhost
export PGPORT=5432
export PGUSER=postgres
export PGPASSWORD=postgres_007%
HOSTNAME=localhost # set it to the actual ip address or host name
createdb gis --host="$HOSTNAME" --encoding="UTF8" --lc-collate="en_GB.UTF-8" --lc-ctype="en_GB.UTF-8" --template=template0

If you get the following error:

ERROR:  invalid locale name: "en_GB.utf8"

then you need to add ‘en_GB.utf8’ locale using the following command:

sudo dpkg-reconfigure locales

And select “en_GB.UTF-8 UTF-8” in the first screen (“Locales to be generated”). Subsequently, restarting the db would be suggested:

sudo service postgresql restart

If you get the following error:

ERROR:  new collation (en_GB.utf8) is incompatible with the collation of the template database (en_US.UTF-8)
HINT:  Use the same collation as in the template database, or use template0 as template.

you need to use template0 for gis:

psql -U postgres -h $HOSTNAME -c "CREATE DATABASE gis ENCODING 'UTF-8' LC_COLLATE 'en_GB.utf8' LC_CTYPE 'en_GB.utf8' TEMPLATE template0"

# alternative command: createdb -E UTF8 -l en_GB.UTF8 -O postgres  -T template0 gis

If you get the following error:

ERROR:  new encoding (UTF8) is incompatible with the encoding of the template database (SQL_ASCII)
HINT:  Use the same encoding as in the template database, or use template0 as template.

(error generally happening with Ubuntu on Windows with WSL), then add also TEMPLATE template0; e.g., use the following command:

psql -U postgres -h $HOSTNAME -c "CREATE DATABASE gis ENCODING 'UTF-8' LC_COLLATE 'en_GB.utf8' LC_CTYPE 'en_GB.utf8' TEMPLATE template0"
# alternative command: createdb -E UTF8 -l en_GB.utf8 -O postgres -T template0 gis

Check to create the DB within a disk partition where enough disk space is available3. If you need to use a different tablespace than the default one, execute the following commands instead of the previous ones (example: the tablespace has location /tmp/db):

sudo mkdir /mnt/db # Suppose this is the tablespace location
sudo chown postgres:postgres /mnt/db
psql -U postgres -h $HOSTNAME -c "CREATE TABLESPACE gists LOCATION '/mnt/db'"
psql -U postgres -h $HOSTNAME -c "ALTER DATABASE gis SET TABLESPACE gists"

Create the postgis and hstore extensions:

psql -U postgres -h $HOSTNAME -c "\connect gis"
psql -U postgres -h $HOSTNAME -d gis -c "CREATE EXTENSION postgis"
psql -U postgres -h $HOSTNAME -d gis -c "CREATE EXTENSION hstore"

If you get the following error

ERROR: could not open extension control file "/usr/share/postgresql/9.3/extension/postgis.control": No such file or directory

then you might be installing PostgreSQL 9.3 (instead of 9.5), for which you should also need:

sudo apt-get install postgis postgresql-9.3-postgis-scripts

Install it and repeat the create extension commands. Notice that PostgreSQL 9.3 is not currently supported by openstreetmap-carto.

Add a user and grant access to gis DB

In order for the application to access the gis database, a DB user with the same name of your UNIX user is needed. Let’s suppose your UNIX ue is tileserver.

psql -d gis -c "create user tileserver;grant all privileges on database gis to postgres;"
psql -d gis -c 'create user "www-data";grant all privileges on database gis to "www-data";'

psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO postgres;'
psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO  postgres;'

Enabling remote access to PostgreSQL

If in different host, to remotely access PostgreSQL, you need to edit pg_hba.conf:

sudo vi /etc/postgresql/*/main/pg_hba.conf

and add the following line:

host    all             all             <your IP set>/<your netmask>             md5

host all all 0.0.0.0/0 md5 is an access control rule that let anybody login in from any address if providing a valid password (md5 keyword).

Then edit postgresql.conf:

sudo vi /etc/postgresql/*/main/postgresql.conf

and set listen_addresses = '*'

Finally, the DB shall be restarted:

sudo /etc/init.d/postgresql restart

Check that the gis database is available. To list all databases defined in PostgreSQL, issue the following command:

psql -U postgres -h $HOSTNAME -c "\l+"

The obtained report should include the gis database, as in the following table:

Name Owner Encoding Collate Ctype Access privileges
gis postgres UTF8 en_US.utf8 en_US.utf8 =Tc/postgres
          postgres=CTc/postgres
          tileserver=CTc/postgres

Tuning the database

The default PostgreSQL settings aren’t great for very large databases like OSM databases. Proper tuning can just about double the performance.

Minimum tuning requirements

Set the postgres user to trust:

sudo vi /etc/postgresql/*/main/pg_hba.conf
# change: local   all             postgres                                peer
# to:     local   all             postgres                                trust

After performing the above change, restart the DB:

sudo service postgresql restart

Run tune-postgis.sh:

export POSTGRES_USER=postgres
export PG_MAINTENANCE_WORK_MEM=256MB
export PG_WORK_MEM=16MB
export psql=psql

cd ~/src
cd openstreetmap-carto
bash scripts/tune-postgis.sh

Whitout setting postgres to trust, the following error occurs: psql: error: FATAL: Peer authentication failed for user "postgres" when running tune-postgis.sh.

To cleanup the data directory and redo again tune-postgis.sh: rm -rf data.

Optional further tuning requirements

The PostgreSQL wiki has a page on database tuning.

Paul Norman’s Blog has an interesting note on optimizing the database, which is used here below.

Default maintenance_work_mem and work_mem settings are far too low for rendering.4: both parameters should be increased for faster data loading and faster queries (index scanning).

Conservative settings for a 4GB VM are work_mem=32MB and maintenance_work_mem=256MB. On a machine with enough memory you could set them as high as work_mem=256MB and maintenance_work_mem=1GB.

Besides, important settings are shared_buffers and the write-ahead-log (wal). There are also some other settings you might want to change specifically for the import.

To edit the PostgreSQL configuration file with vi editor:

sudo vi /etc/postgresql/*/main/postgresql.conf

and if you are running PostgreSQL 9.3 (not supported):

sudo vi /etc/postgresql/9.3/main/postgresql.conf

Suggested minimum settings:

shared_buffers = 128MB
min_wal_size = 1GB
max_wal_size = 2GB
work_mem = 32MB # check comments for better tuning
maintenance_work_mem = 256MB
autovacuum = off
fsync = off

The latter two ones allow a faster import: the first turns off auto-vacuum during the import and allows you to run a vacuum at the end; the second introduces data corruption in case of a power outage and is dangerous. If you have a power outage while importing the data, you will have to drop the data from the database and re-import, but it’s faster. Just remember to change these settings back after importing. fsync has no effect on query times once the data is loaded.

The PostgreSQL tuning adopted by OpenStreetMap can be found in the PostgreSQL Chef Cookbook: the specific PostgreSQL tuning for the OpenStreetMap tile servers is reported in the related Tileserver Chef configuration.

For a dev&test installation on a system with 16GB of RAM, the suggested settings are the following5:

shared_buffers = 2GB
work_mem = 256MB
maintenance_work_mem = 1GB
wal_level = minimal
synchronous_commit = off
min_wal_size = 1GB
max_wal_size = 2GB
checkpoint_segments = 60
checkpoint_timeout = 15min
checkpoint_completion_target = 0.9
default_statistics_target = 1000
autovacuum = off
fsync = off

default_statistics_target can be even increased to 10000.

If performing database updates, run ANALYZE periodically.

To stop and start the database:

sudo /etc/init.d/postgresql stop

sudo /etc/init.d/postgresql start

You may get an error and need to increase the shared memory size. Edit /etc/sysctl.d/30-postgresql-shm.conf and run sudo sysctl -p /etc/sysctl.d/30-postgresql-shm.conf. A parameter like kernel.shmmax=17179869184 and kernel.shmall=4194304 could be appropriate for a 16GB segment size.6

To manage and maintain the configuration of the servers run by OpenStreetMap, the Chef configuration management tool is used.

The configuration adopted for PostgreSQL is postgresql/attributes/default.rb.

Install Osm2pgsql

Osm2pgsql is an OpenStreetMap specific software used to load the OSM data into the PostGIS database.

The default packaged versions of Osm2pgsql are 0.88.1-1 on Ubuntu 16.04 LTS and 0.96.0 on Ubuntu 18.04 LTS. Nevertheless, more recent versions are suggested, available at the OpenStreetMap Osmadmins PPA or compiling the software from sources.

To install osm2pgsql:

sudo apt install -y osm2pgsql

To install Osm2pgsql from Osmadmins PPA:

sudo add-apt-repository -y ppa:osmadmins/ppa
apt-key adv --keyserver keyserver.ubuntu.com --recv A438A16C88C6BE41CB1616B8D57F48750AC4F2CB
sudo apt-get update
sudo apt-get install -y osm2pgsql

Go to Get an OpenStreetMap data extract.

Generate Osm2pgsql from sources

This alternative installation procedure generates the most updated executable by compiling the sources.

Install Needed dependencies:

sudo apt-get install -y make cmake g++ libboost-dev libboost-system-dev \
  libboost-filesystem-dev libexpat1-dev zlib1g-dev \
  libbz2-dev libpq-dev libgeos-dev libgeos++-dev libproj-dev lua5.2 \
  liblua5.2-dev

Download osm2pgsql:

mkdir -p ~/src ; cd ~/src
git clone git://github.com/openstreetmap/osm2pgsql.git

Prepare for compiling, compile and install:

cd osm2pgsql
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make
sudo make install
cd

Get an OpenStreetMap data extract

You need to download an appropriate .osm or .pbf file to be subsequently loaded into the previously created PostGIS instance via osm2pgsql.

There are many ways to download the OSM data.

The reference is Planet OSM.

It’s probably easiest to grab an PBF of OSM data from geofabrik.

Also, BBBike.org provides extracts of more than 200 cities and regions world-wide in different formats.

Examples:

  • Map data of the whole planet (32G):

    wget -c https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf
    
  • Map data of Great Britain (847M):

    wget -c https://download.geofabrik.de/europe/great-britain-latest.osm.pbf
    
  • For just Liechtenstein:

    wget https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf
    wget https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf.md5
    
    # Optionally, the following will xheck that the download wasn't corrupted:
    md5sum -c liechtenstein-latest.osm.pbf.md5
    

Another method to download data is directly with your browser. Check this page.

Alternatively, JOSM can be used (Select the area to download the OSM data: JOSM menu, File, Download From OSM; tab Slippy map; drag the map with the right mouse button, zoom with the mouse wheel or Ctrl + arrow keys; drag a box with the left mouse button to select an area to download. The Continuous Download plugin is also suggested. When the desired region is locally available, select File, Save As, <filename>.osm. Give it a valid file name and check also the appropriate directory where this file is saved.

In all cases, avoid using too small areas.

OpenStreetMap is open data. OSM’s license is Open Database License.

Load data to PostGIS

The osm2pgsql documentation reports all needed information to use this ETL tool, including related command line options.

osm2pgsql uses overcommit like many scientific and large data applications, which requires adjusting a kernel setting:

sudo sysctl -w vm.overcommit_memory=1

To load data from an .osm or .pbf file to PostGIS, issue the following:

cd ~/src
cd openstreetmap-carto

export OSM2PGSQL_CACHE=${OSM2PGSQL_CACHE:-512}
export OSM2PGSQL_NUMPROC=${OSM2PGSQL_NUMPROC:-1}
export OSM2PGSQL_DATAFILE=${OSM2PGSQL_DATAFILE:-data.osm.pbf}

osm2pgsql \
--cache $OSM2PGSQL_CACHE \
--number-processes $OSM2PGSQL_NUMPROC \
--hstore \
--multi-geometry \
--database gis \
--slim \
--drop \
--style openstreetmap-carto.style \
--tag-transform-script openstreetmap-carto.lua \
[.osm or .pbf file]

With available memory, set export OSM2PGSQL_CACHE=2500; it allocates 2.5 GB of memory to the import process.

Option --create loads data into an empty database rather than trying to append to an existing one.

Relaying to OSM2PGSQL_NUMPROC, if you have more cores available, you can set it accordingly.

The osm2pgsql manual describes usage and all options in detail.

Go to the next step.

If using a different server:

cd ~/src
cd openstreetmap-carto
HOSTNAME=localhost # set it to the actual ip address or host name
osm2pgsql -s -C 300 -c -G --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua -d gis -H $HOSTNAME -U postgres [.osm or .pbf file]

Notice that the suggested process adopts the -s (--slim option), which uses temporary tables, so running it takes more diskspace (and is very slow), while less RAM memory is used. You might add --drop option with -s (--slim), to also drop temporary tables after import, otherwise you will also find the temporary tables nodes, ways, and rels (these tables started out as pure “helper” tables for memory-poor systems, but today they are widely used because they are also a prerequisite for updates).

If everything is ok, you can go to the next step.

Notice that the following elements are used:

  • hstore
  • the openstreetmap-carto.style
  • the openstreetmap-carto.lua LUA script
  • gis DB name

Depending on the input file size, the osm2pgsql command might take very long. An interesting page related to Osm2pgsql benchmarks associates sizing of hw/sw systems with related figures to import OpenStreetMap data.

Note: if you get the following error:

node_changed_mark failed: ERROR:  prepared statement "node_changed_mark" does not exist

do the following command on your original.osm:

sed "s/action='modify' //" < original.osm | > fixedfile.osm

Then process fixedfile.osm.

If you get errors like this one:

Error reading style file line 79 (fields=4)
flag 'phstore' is invalid in non-hstore mode
Error occurred, cleaning up

or this one:

Postgis Plugin: ERROR:  column "tags" does not exist
LINE 8: ...ASE WHEN "natural" IN ('mud') THEN "natural" ELSE tags->'wet...

then you need to enable hstore extension to the db with CREATE EXTENSION hstore; and also add the –hstore flag to osm2pgsql. Enabling hstore extension and using it with osm2pgsql will fix those errors.

Create the data folder

At least 18 GB HD and appropriate RAM/swap is needed for this step (24 GB HD is better). 8 GB HD will not be enough. With 1 GB RAM, configuring a swap is mandatory.

python3 -m pip install psycopg2-binary

cd ~/src
cd openstreetmap-carto
scripts/get-external-data.py

To cleanup the get-external-data.py procedure and restart from scratch, remove the data directory (rm -r data).

Configure a swap to prevent the following message:

INFO:root:Checking table water_polygons
Killed

The way shapefiles are loaded by the OpenStreetMap tile servers is reported in the related Chef configuration.

Read scripted download for further information.

Create indexes and grant users

Create partial indexes to speed up the queries included in project.mml and grant access to all gis tables to avoid renderd errors when accessing tables with user tileserver.

  • Add the partial geometry indexes indicated by openstreetmap-carto7 to provide effective improvement to the queries:

    cd ~/src
    cd openstreetmap-carto
    HOSTNAME=localhost # set it to the actual ip address or host name
    psql -d gis -f indexes.sql
    

    Alternative mode:

    cd ~/src
    cd openstreetmap-carto
    scripts/indexes.py | psql -d gis
    

    If using a different host:

    HOSTNAME=localhost # set it to the actual ip address or host name
    cd ~/src
    cd openstreetmap-carto
    scripts/indexes.py | psql -U postgres -h $HOSTNAME -d gis
    

    Alternative mode with a different host:

    HOSTNAME=localhost # set it to the actual ip address or host name
    psql -U postgres -h $HOSTNAME -d gis -f indexes.sql
    
  • Create PostgreSQL user “tileserver” (if not yet existing) and grant rights to all gis db tables for “tileserver” user and for all logged users:

psql -d gis <<\eof
REVOKE CONNECT ON DATABASE gis FROM PUBLIC;
GRANT CONNECT ON DATABASE gis TO "www-data";
GRANT CONNECT ON DATABASE gis TO "tileserver";
eof

To list all tables available in the gis database, issue the following command:

psql -d gis -c "\dt+"

or:

psql -U postgres -h $HOSTNAME -d gis -c "\dt+"

The database shall include the rels, ways and nodes tables (created with the --slim mode of osm2pgsql) in order to allow updates.

In the following example of output, the --slim mode of osm2pgsql was used:

Schema Name Type Owner
public planet_osm_line table postgres
public planet_osm_nodes table postgres
public planet_osm_point table postgres
public planet_osm_polygon table postgres
public planet_osm_rels table postgres
public planet_osm_roads table postgres
public planet_osm_ways table postgres
public spatial_ref_sys table postgres

In fact, the tables planet_osm_rels, planet_osm_ways, planet_osm_nodes are available, as described in the Database Layout of Pgsql.

Check The OpenStreetMap data model at Mapbox for further details.

Read custom indexes for further information.

Install Kosmtik plugins

Openstreetmap-carto suggests installing some Kosmtik plugins:

Case of global installation (plugins management through kosmtik is deprecated, one should install the normal way):


sudo npm install -g leaflet
sudo npm install -g leaflet-editinosm
sudo npm install -g leaflet.photon
sudo npm install -g osmtogeojson

# Plugins that should work:
sudo npm install -g git+https://github.com/kosmtik/kosmtik-overpass-layer # Add overpass layers to your Kosmtik project
sudo npm install -g git+https://github.com/kosmtik/kosmtik-fetch-remote # Kosmtik plugin to fetch remote layer data
sudo npm install -g git+https://github.com/kosmtik/kosmtik-overlay # Add an overlay in Kosmtik
sudo npm install -g git+https://github.com/kosmtik/kosmtik-map-compare # Add a map to compare side-by-side with your work in Kosmtik
sudo npm install -g git+https://github.com/kosmtik/kosmtik-osm-data-overlay # Display OSM data on top of your map in Kosmtik.
sudo npm install -g git+https://github.com/kosmtik/kosmtik-tiles-export # Export tiles tree of your Kosmtik map
sudo npm install -g git+https://github.com/kosmtik/kosmtik-deploy # Deploy your Kosmtik project on a remote server
sudo npm install -g git+https://github.com/kosmtik/kosmtik-mapnik-reference # Browse Mapnik and CartoCSS reference in Kosmtik; needs  --mapnik-version 3.0.20
sudo npm install -g git+https://github.com/kosmtik/kosmtik-open-in-josm # Add a control in Kosmtik to open your favorite OSM editor with the current
sudo npm install -g git+https://github.com/kosmtik/kosmtik-geojson-overlay # Show a geojson overlay on top of your Kosmtik map
sudo npm install -g git+https://github.com/kosmtik/kosmtik-place-search # Search control for Kosmtik

# Plugins that might not work:
sudo npm install -g git+https://github.com/kosmtik/kosmtik-magnacarto # Magnacarto renderer support for Kosmtik
sudo npm install -g git+https://github.com/kosmtik/kosmtik-mbtiles-export # Export your Kosmtik project in MBTiles
sudo npm install -g git+https://github.com/kosmtik/kosmtik-mbtiles-source # Use an MBTiles file as a source for a vector tile based probject in kosmtik
sudo npm install -g git+https://github.com/kosmtik/kosmtik-places # Bookmark places in Kosmtik

Case of local installation:

cd ~/src/kosmtik

npm install leaflet
npm install leaflet-editinosm
npm install leaflet.photon
npm install osmtogeojson

# Plugins that should work:
npm install git+https://github.com/kosmtik/kosmtik-overpass-layer # Add overpass layers to your Kosmtik project
npm install git+https://github.com/kosmtik/kosmtik-fetch-remote # Kosmtik plugin to fetch remote layer data
npm install git+https://github.com/kosmtik/kosmtik-overlay # Add an overlay in Kosmtik
npm install git+https://github.com/kosmtik/kosmtik-map-compare # Add a map to compare side-by-side with your work in Kosmtik
npm install git+https://github.com/kosmtik/kosmtik-osm-data-overlay # Display OSM data on top of your map in Kosmtik.
npm install git+https://github.com/kosmtik/kosmtik-tiles-export # Export tiles tree of your Kosmtik map
npm install git+https://github.com/kosmtik/kosmtik-deploy # Deploy your Kosmtik project on a remote server
npm install git+https://github.com/kosmtik/kosmtik-mapnik-reference # Browse Mapnik and CartoCSS reference in Kosmtik; needs  --mapnik-version 3.0.20
npm install git+https://github.com/kosmtik/kosmtik-open-in-josm # Add a control in Kosmtik to open your favorite OSM editor with the current
npm install git+https://github.com/kosmtik/kosmtik-geojson-overlay # Show a geojson overlay on top of your Kosmtik map
npm install git+https://github.com/kosmtik/kosmtik-place-search # Search control for Kosmtik

# Plugins that might not work:
npm install git+https://github.com/kosmtik/kosmtik-mbtiles-export # Export your Kosmtik project in MBTiles
npm install git+https://github.com/kosmtik/kosmtik-magnacarto # Magnacarto renderer support for Kosmtik
npm install git+https://github.com/kosmtik/kosmtik-mbtiles-source # Use an MBTiles file as a source for a vector tile based probject in kosmtik
npm install git+https://github.com/kosmtik/kosmtik-places # Bookmark places in Kosmtik

cd ~/src

Activate plugins:

mkdir -p /home/$USER/.config
cat > /home/$USER/.config/kosmtik.yml <<\eof
plugins:
  # Plugins that should work:
  - kosmtik-fetch-remote
  - kosmtik-geojson-overlay
  - kosmtik-map-compare
  - kosmtik-overlay
  - kosmtik-overpass-layer
  - kosmtik-tiles-export
  - kosmtik-deploy
  - kosmtik-mapnik-reference # this needs --mapnik-version 3.0.20
  - kosmtik-open-in-josm # this only works locally
  - kosmtik-osm-data-overlay
  - kosmtik-place-search
  # Plugins that might not work:
  #- kosmtik-magnacarto
  #- kosmtik-mbtiles-export
  #- kosmtik-places
  #- kosmtik-mbtiles-source
eof

To show plaugins and kosmtik version:

kosmtik version

Full list of plugins can be found within the Kosmtik GitHub repository.

Start Kosmtik

Run Kosmtik:

in case of installation with the -g option:

cd ~/src
cd openstreetmap-carto
kosmtik serve project.mml --host 0.0.0.0 --mapnik-version 3.0.20

or also, in case of local installation:

~/src/kosmtik/index.js serve ~/src/openstreetmap-carto/project.mml --localconfig ~/src/openstreetmap-carto/localconfig.json --host 0.0.0.0

Read Usage for further information.

Notice that --host 0.0.0.0 is needed to access Kosmtik installed on a remote server (not necessary when doing http://localhost:6789).

Access the map from your browser

With your browser, access the map through http://localhost:6789

Notice that Https will not work (use http instead).

Note: the following Kosmtik warnings can be ignored:

Mapnik LOG> ...: warning: unable to find face-name 'unifont Medium' in FontSet 'fontset-0'
Mapnik LOG> ...: warning: unable to find face-name 'unifont Medium' in FontSet 'fontset-1'
Mapnik LOG> ...: warning: unable to find face-name 'unifont Medium' in FontSet 'fontset-2'

To cosmetically remove these warinings, install the Old unifont Medium font.

Accessing the database and rendering images is often a slow process (mainly depending on the amount of data to be managed, but also on the server performance and on the network), so give many seconds to Kosmtik to output or refresh the map.

Edit the stylesheets

  • Use your own text editor, e.g., vi
  • Change the *.mss files
  • Kosmtik automatically updates rendering upon file change
  • View the changes in Kosmtik
  1. pnorman comment on hinted fonts 

  2. Is lowercase “unifont” needed? 

  3. Import error: could not extend file 

  4. Information taken from switch2osm

  5. Most reliable way to import large dataset with osm2psql 

  6. Information from Paul Norman’s Blog

  7. .travis.yml applies the custom indexes via psql -1Xq -v ON_ERROR_STOP=1 -d gis -f indexes.sql. Notice that indexes.sql shall be kept up to date with indexes.py and this is also checked by .travis.yml. 


Please, avoid using Disqus below to notify issues on this page, just use it for general comments. Create a GitHub issue instead. This is the most effective way to track problems.
comments powered by Disqus