OpenStreetMap Carto Tutorials - OpenStreetMap Carto documentation     Site map    

Installing TileMill and OpenStreetMap-Carto on Ubuntu

Installing TileMill and OpenStreetMap-Carto on Ubuntu

Introduction

The following step-by-step procedure can be used to install a development environment of openstreetmap-carto exploiting TileMill on an Ubuntu PC.1

Tilemill was the original tool for the development of the openstreetmap-carto style. It moved out of the Mapbox profile and shifted to an open source community-driven organization, with its own organization and contributor model: tilemill-project. At the moment, the suggested tool for the autohoring of OpenStreetMap stylesheets is Kosmtik. TileMill is not officially supported.

Additional information:

A PostGIS database is needed and can be installed locally (suggested) or remotely (might be slow, depending on the network).

The goal for this procedure is to use Ubuntu packages and official PPAs whenever possible.

We consider using Ubuntu 16.04.1 LTS Xenial Xerus, Ubuntu 15.4 Vivid Vervet or Ubuntu 14.04.3 LTS Trusty Tahr (other versions should work). All should be 64-bit computing architecture.

Other distributions like Debian might require changes and are not tested.

This procedure is updated to the version of OpenStreetMap Carto available at the time of writing. To get the correct installation procedure, the INSTALL history should be checked, considering that the OpenStreetMap Carto maintainers use to keep the INSTALL page updated. Check also the README changelog.

General setup for Ubuntu

Update Ubuntu

Make sure your Ubuntu system is fully up-to-date:

lsb_release -a

Previous command returns the Ubuntu version.

To update the system:

sudo apt-get update
sudo apt-get -y upgrade

If on a brand new system you also want to do sudo apt-get dist-upgrade && sudo shutdown -r now.

Install essential tools

sudo apt-get -y install curl unzip gdal-bin tar wget bzip2 build-essential clang

For the subsequent installation steps, we suppose that cd defaults to your home directory.

Configure a swap

Importing and managing map data takes a lot of RAM and a swap is generally needed.

To check whether a swap partition is already configured on your system, use one of the following two commands:

  • Reports the swap usage summary (no output means missing swap):

    swapon -s
    
  • Display amount of free and used memory in the system (check the line specifying Swap):

    free -h
    

If you do not have an active swap partition, especially if your physical memory is small, you should add a swap file. First we use fallocate command to create a file. For example, create a file named swapfile with 2G capacity in root file system:

sudo fallocate -l 2G /swapfile

Then make sure only root can read and write to it.

sudo chmod 600 /swapfile

Format it to swap:

sudo mkswap /swapfile

Enable the swap file

sudo swapon /swapfile

The Operating System tuning adopted by the OpenStreetMap tile servers can be found in the related Chef configuration.

Check usage of English locale

Run locale to list what locales are currently defined for the current user account:

locale

To set the en_US locale:

export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

The exported variables can be put to the file /etc/environment.

New locales can also be generated by issuing:

sudo locale-gen en_US en_US.UTF-8
sudo dpkg-reconfigure locales

Install Git

Git should be already installed on Ubuntu 16.04.

git --version # to verify whether git is already installed
sudo apt-get install -y git

Install Node.js

To verify whether Node.js is already installed:

nodejs --version

nodejs-legacy might be needed for TileMill (at least required with Ubuntu 16.04 at the time of writing).

sudo apt-get install -y nodejs-legacy npm

The following error when running TileMill is related to compatibility issues with nodejs and should be fixed by installing nodejs-legacy.

npm ERR! Failed at the mapnik@3.5.13 install script 'node-pre-gyp install --fallback-to-build'.
npm ERR! Make sure you have the latest version of node.js and npm installed.`

Installing the latest version of nodejs is not suggested at the moment. If you really want to do it:

sudo apt-get install -y nodejs npm # this package might not support TileMill, see notes in this paragraph

Read nodejs for further information.

Install TileMill

Optional elements (needed for the topcube module related to the client user interface, not needed if TileMill will only be run in server mode):

# Install the topcube module, UI prerequisite to TileMill:
sudo apt-get install -y libgtk2.0-dev libwebkit-dev libwebkitgtk-dev

Installation of TileMill:

test -d ~/src || mkdir  ~/src ; cd ~/src
git clone https://github.com/tilemill-project/tilemill.git
cd ~/src/tilemill
npm install

Test TileMill

cd ~/src/tilemill
npm test # you can also run TileMill to test: see "Start TileMill" below.

See Start TileMill.

Some test might not pass (this does not mean that the installation is necessarily failed)

Notice that, when running npm test, an error like the following indicates that your system does not have a modern enough libstdc++/gcc-base toolchain:

Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.20 not found (required by /node_modules/osrm/lib/binding/osrm.node)

If you are running Ubuntu older than 16.04 you can easily upgrade your libstdc++ version like:

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update -y
sudo apt-get install -y libstdc++-5-dev

Read node-mapnik for further information.

Python installation

Check that Python is installed:

python -V
python3 -V

Otherwise Python needs to be installed.

Install Yaml and Package Manager for Python

This is necessary in order to run OpenStreetMap-Carto scripts/indexes.

sudo apt-get install -y python-yaml

pip -V # to verify whether pip is already installed
sudo apt-get install -y python-pip

Install Mapnik Utilities

The Mapnik Utilities package includes shapeindex.

sudo apt-get install -y mapnik-utils

Install openstreetmap-carto

cd /home/ubuntu/Documents/MapBox/project # if this directory is missing, start TileMill to create it
git clone https://github.com/gravitystorm/openstreetmap-carto.git
cd openstreetmap-carto

Read installation notes for further information.

Install the fonts needed by openstreetmap-carto

Currently Noto fonts are used.

To install them (except Noto Emoji Regular):

sudo apt-get install -y fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted fonts-hanazono ttf-unifont

Installation of Noto Emoji Regular:

cd ~/src
git clone https://github.com/googlei18n/noto-emoji.git
sudo cp noto-emoji/fonts/NotoColorEmoji.ttf noto-emoji/fonts/NotoEmoji-Regular.ttf /usr/share/fonts/truetype/noto
sudo fc-cache -fv
sudo apt install fontconfig
fc-list
fc-list | grep Emoji
cd openstreetmap-carto

DejaVu Sans is used as an optional fallback font for systems without Noto Sans. If all the Noto fonts are installed, it should never be used.

sudo apt-get install -y fonts-dejavu-core

Read font notes for further information.

Create the data folder

cd /home/ubuntu/Documents/MapBox/project # if this directory is missing, start TileMill to create it
cd openstreetmap-carto
scripts/get-shapefiles.py

The actual shapefiles loaded by the OpenStreetMap tile servers are reported in the related Chef configuration.

Read scripted download for further information.

Set the environment variables

export PGHOST=localhost
export PGPORT=5432
export PGUSER=postgres
export PGPASSWORD=postgres_007%

Configure the firewall

If you are preparing a remote virtual machine, configure the firewall to allow remote access to the local port 20009.

If you run a cloud based VM, also the VM itself shall be set to open this port.

Install PostgreSQL and PostGIS

PostgreSQL is a relational database, and PostGIS is its spatial extender, which allows you to store geographic objects like map data in it; it serves a similar function to ESRI’s SDE or Oracle’s Spatial extension. PostgreSQL + PostGIS are used for a wide variety of features such as rendering maps, geocoding, and analysis.

Currently the tested versions for OpenstreetMap Carto are PostgreSQL 9.5 and PostGIS 2.2:

Also older PostgreSQL version should be suitable.

On Ubuntu there are pre-packaged versions of both postgis and postgresql, so these can simply be installed via the Ubuntu package manager.

sudo apt-get update
sudo apt-get install -y postgresql postgis pgadmin3 postgresql-contrib

Note: used PostgeSQL port is 5432 (default).

A user named postgres will be created during the installation process.

Set the password for the postgres user

sudo -u postgres psql postgres
\password postgres

Alternative procedure (useful if you get authentication issues with the previous one):

sudo su -
sudo -i -u postgres
psql postgres
\password postgres

Enter the following password twice: postgres_007%

This is just an example of password, you can use the one you prefer.

After entering the password, exit from psql with:

\q

With the second procedure, also isssue:

exit # from 'sudo -i -u postgres'
exit # from 'sudo su -'

Create the PostGIS instance

Now you need to create a postgis database. The defaults of various programs including openstreetmap-carto (ref. project.mml) assume the database is called gis. You need to set up PostGIS on the PostgreSQL database.

export PGPASSWORD=postgres_007%
HOSTNAME=localhost # set it to the actual ip address or host name
psql -U postgres -h $HOSTNAME -c "create database gis" # alternative command: createdb -E UTF8 -O postgres gis
psql -U postgres -h $HOSTNAME -c "\connect gis"
psql -U postgres -h $HOSTNAME -d gis -c "CREATE EXTENSION postgis"
psql -U postgres -h $HOSTNAME -d gis -c "CREATE EXTENSION hstore"

The character encoding scheme to be used in the database is UTF8.

If you get the following error

ERROR: could not open extension control file "/usr/share/postgresql/9.3/extension/postgis.control": No such file or directory

then you might be installing PostgreSQL 9.3, for which you should also need:

sudo apt-get install postgis postgresql-9.3-postgis-scripts

Install it and repeat the create extension commands.

If you need to use a different tablespace than the default one, execute the following commands instead of the previous ones (example: the tablespace has location /tmp/db):

export PGPASSWORD=postgres_007%
HOSTNAME=localhost # set it to the actual ip address or host name
sudo mkdir /mnt/db # Suppose this is the tablespace location
sudo chown postgres:postgres /mnt/db
psql -U postgres -h $HOSTNAME -c "CREATE TABLESPACE gists LOCATION '/mnt/db'"
psql -U postgres -h $HOSTNAME -c "CREATE DATABASE gis TABLESPACE gists"
psql -U postgres -h $HOSTNAME -c "\connect gis"
psql -U postgres -h $HOSTNAME -d gis -c "CREATE EXTENSION postgis"
psql -U postgres -h $HOSTNAME -d gis -c "CREATE EXTENSION hstore"

Add a user and grant access to gis DB

In order for the application to access the gis database, a DB user with the same name of your UNIX user is needed. Let’s suppose your UNIX ue is tileserver.

sudo su -
sudo -i -u postgres
createuser tileserver
psql
grant all privileges on database gis to tileserver;
\q
exit
exit

Enabling remote access to PostgreSQL

To remotely access PostgreSQL, you need to edit pg_hba.conf:

sudo vi /etc/postgresql/9.5/main/pg_hba.conf

and add the following line:

host    all             all             <your IP set>/<your netmask>             md5

host all all 0.0.0.0/0 md5 is an access control rule that let anybody login in from any address if providing a valid password (md5 keyword).

Then edit postgresql.conf:

sudo vi /etc/postgresql/9.5/main/postgresql.conf

and set listen_addresses = '*'

Finally, the DB shall be restarted:

sudo /etc/init.d/postgresql restart

Tuning the database

The default PostgreSQL settings aren’t great for very large databases like OSM databases. Proper tuning can just about double the performance.

The PostgreSQL wiki has a page on database tuning.

Paul Norman’s Blog has an interesting note on optimizing the database, which is used here below.

Adjusting maintenance_work_mem and work_mem are probably enough on a development or testing machine.2: both parameters should be increased for faster data loading and faster queries while rendering respectively.

Conservative settings for a 2GB VM are work_mem=16MB and maintenance_work_mem=128MB. On a machine with enough memory you could set them as high as work_mem=128MB and maintenance_work_mem=1GB.

Besides, important settings are shared_buffers and the write-ahead-log (wal). There are also some other settings you might want to change specifically for the import.

To edit the PostgreSQL configuration file with vi editor:

sudo vi /etc/postgresql/9.5/main/postgresql.conf

and if you are running PostgreSQL 9.3:

sudo vi /etc/postgresql/9.3/main/postgresql.conf

Suggested settings:

shared_buffers = 128MB
min_wal_size = 1GB
max_wal_size = 2GB
maintenance_work_mem = 256MB
autovacuum = off
fsync = off

The latter two ones allow a faster import: the first turns off auto-vacuum during the import and allows you to run a vacuum at the end; the second introduces data corruption in case of a power outage and is dangerous. If you have a power outage while importing the data, you will have to drop the data from the database and re-import, but it’s faster. Just remember to change these settings back after importing. fsync has no effect on query times once the data is loaded.

The PostgreSQL tuning adopted by OpenStreetMap can be found in the PostgreSQL Chef Cookbook: the specific PostgreSQL tuning for the OpenStreetMap tile servers is reported in the related Tileserver Chef configuration.

For a dev&test installation on a system with 16GB of RAM, the suggested settings are the following:

shared_buffers = 2GB
work_mem = 128MB
maintenance_work_mem = 1GB
wal_level = minimal
synchronous_commit = off
min_wal_size = 1GB
max_wal_size = 2GB
checkpoint_timeout = 15min
checkpoint_completion_target = 0.9
default_statistics_target = 1000
autovacuum = off
fsync = off

To stop and start the database:

sudo /etc/init.d/postgresql stop

sudo /etc/init.d/postgresql start

You may get an error and need to increase the shared memory size. Edit /etc/sysctl.d/30-postgresql-shm.conf and run sudo sysctl -p /etc/sysctl.d/30-postgresql-shm.conf. A parameter like kernel.shmmax=17179869184 and kernel.shmall=4194304 could be appropriate for a 16GB segment size.3

To manage and maintain the configuration of the servers run by OpenStreetMap, the Chef configuration management tool is used.

The configuration adopted for PostgreSQL is postgresql/attributes/default.rb.

Install Osm2pgsql

Osm2pgsql is an OpenStreetMap specific software used to load the OSM data into the PostGIS database.

To install osm2pgsql:

sudo apt-get install -y osm2pgsql

Go to Get an OpenStreetMap data extract.

Alternative installation procedure

This alternative installation procedure generates the most updated executable by compiling the sources.

# Needed dependencies
sudo apt-get install -y make cmake g++ libboost-dev libboost-system-dev \
  libboost-filesystem-dev libexpat1-dev zlib1g-dev \
  libbz2-dev libpq-dev libgeos-dev libgeos++-dev libproj-dev lua5.2 \
  liblua5.2-dev

# Download osm2pgsql
cd /tmp
git clone git://github.com/openstreetmap/osm2pgsql.git 

# Prepare for compiling
cd osm2pgsql
mkdir build && cd build
cmake ..

# Compile
make

# Install
sudo make install

# Clean-out temporary files
cd ../..
rm -rf osm2pgsql

Get an OpenStreetMap data extract

You need to download an appropriate .osm or .pbf file to be subsequently loaded into the previously created PostGIS instance via osm2pgsql.

There are many ways to download the OSM data.

The reference is Planet OSM.

It’s probably easiest to grab an PBF of OSM data from Mapzen or geofabrik.

Also, BBBike.org provides extracts of more than 200 cities and regions world-wide in different formats.

Examples:

  • Map data of the whole planet (32G):

    wget -c http://planet.openstreetmap.org/pbf/planet-latest.osm.pbf
    
  • Map data of Great Britain (847M):

    wget -c http://download.geofabrik.de/europe/great-britain-latest.osm.pbf
    
  • Map data of Lombardy (279M):

    wget -c http://osm-estratti.wmflabs.org/estratti/regioni/pbf/03---Lombardia.pbf
    
  • For just Liechtenstein:

    wget http://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf.md5
    wget http://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf
    md5sum -c liechtenstein-latest.osm.pbf.md5 # Check that the download wasn't corrupted
    

Another method to download data is directly with your browser. Check this page.

Alternatively, JOSM can be used (Select the area to download the OSM data: JOSM menu, File, Download From OSM; tab Slippy map; drag the map with the right mouse button, zoom with the mouse wheel or Ctrl + arrow keys; drag a box with the left mouse button to select an area to download. The Continuous Download plugin is also suggested. When the desired region is locally available, select File, Save As, <filename>.osm. Give it a valid file name and check also the appropriate directory where this file is saved.

In all cases, avoid using too small areas.

OpenStreetMap is open data. OSM’s license is Open Database License.

Load data to PostGIS

The osm2pgsql documentation reports all needed information to use this ETL tool, including related command line options.

osm2pgsql uses overcommit like many scientific and large data applications, which requires adjusting a kernel setting:

sudo sysctl -w vm.overcommit_memory=1

To load data from an .osm or .pbf file to PostGIS, issue the following:

cd /home/ubuntu/Documents/MapBox/project # if this directory is missing, start TileMill to create it
cd openstreetmap-carto
HOSTNAME=localhost # set it to the actual ip address or host name
osm2pgsql -s -C 300 -c -G --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua -d gis -H $HOSTNAME -U postgres [.osm or .pbf file]

If everything is ok, you can go to Create indexes.

Notice that the following elements are used:

  • hstore
  • the openstreetmap-carto.style
  • the openstreetmap-carto.lua LUA script
  • gis DB name

Depending on the input file size, the osm2pgsql command might take very long.

Note: if you get the following error:

` node_changed_mark failed: ERROR: prepared statement “node_changed_mark” does not exist

do the following command on your original.osm:

sed "s/action='modify' //" < original.osm | > fixedfile.osm

Then process fixedfile.osm.

If you get the following error:

Error reading style file line 79 (fields=4)
flag 'phstore' is invalid in non-hstore mode
Error occurred, cleaning up

then you need to add the hstore flag to osm2pgsql:

osm2pgsql -s -C 300 -c -G --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua -d gis -H $HOSTNAME -U postgres [.osm or .pbf file]

Create indexes

Add the indexes indicated by openstreetmap-carto:

HOSTNAME=localhost # set it to the actual ip address or host name
cd /home/ubuntu/Documents/MapBox/project # if this directory is missing, start TileMill to create it
cd openstreetmap-carto
scripts/indexes.py | psql -U postgres -h $HOSTNAME -d gis

Read custom indexes for further information.

Start TileMill

cd ~/src/tilemill
./index.js --server=true --listenHost=0.0.0.0

Notice that --server=true is only needed in case of server based startup, to avoid the local user interface. Besides, --listenHost=0.0.0.0 is needed to access TileMill installed on a remote server (not necessary when doing http://localhost:20009).

An error message like role … does not exist… might be possibly related to missing environment variables PGUSER, PGPASSWORD, etc. Set these variables and restart TileMill.

The application needs a few seconds to start, so be patient.

Select project Openstreetmap Carto

Give TileMill the time to render the map: accessing the database and rendering images is often a slow process (mainly depending on the amount of data to be managed, but also on the server performance and on the network), so give many seconds to TileMill to output or refresh the map.

Zoom out to the entire world shape (zoom level 1 or to some close zooms like zoom 4), then progressively zoom into the region where you downloaded the map data. You might use the double click and wait for the next zoom level to appear.

You shouldn’t use the text editor built-in to TileMill. Instead, hide the right pane and use an external text editor.

TileMill automatically refreshes the rendering upon any file change, including all .mss and project.mml.

Access the map from your browser

With your browser, access the map through http://localhost:20009

To directly access the project: http://127.0.0.1:20009/#/project/openstreetmap-carto

Notice that Https will not work (use http instead).

TileMill documentation

Edit the stylesheets

  • Use your own text editor, e.g., vi
  • Change the *.mss files
  • TileMill automatically updates rendering upon file change
  • View the changes in TileMill

Footnotes

  1. Part of the documentation is taken from Openstreetmap-carto Provide installation script #657 and from TileMill

  2. Information taken from switch2osm

  3. Information from Paul Norman’s Blog


Please, avoid using Disqus below to notify issues on this page, just use it for general comments. Create a GitHub issue instead. This is the most effective way to track problems.
comments powered by Disqus