Hosting static OSM vector tiles on object storage

Carto, OpenStreetMap contributors

Need a map for your project? You could just use Google Maps, but their styling options are limited and it's expensive at scale. Various providers offer raster tiles in more interesting styles based on OpenStreetMap, but you're still limited to their designs. Vector tiles based on OpenStreetMap offer complete stylistic freedom, as they only contain the raw geographic data, while the styling is entirely up to your client. A few providers also host vector tiles.

Beyond the free tiers offered for small scale hobby projects, costs for hosted tiles of any kind usually rise quite steeply. But actually, map tiles are pretty static. Could we pre-generate all the tiles for an area and just host them on any cheap object storage?

Yes, this is entirely possible if not entirely well documented, and in this post we're going through all the steps needed to host your own vector tiles on Cloudflare R2, complete with a Github Actions workflow to periodically regenerate the tiles from the latest OpenStreetMap data.

Overview

Generally, there are a few steps involved:

  1. Obtain an extract of OpenStreetMap data (.osm.pbf file)
  2. Process the extract into an .mbtiles file containing the vector tile data
  3. Export the .mbtiles file into an z-x-y hierarchy of static Mapbox Vector Tiles (.pbf/.mvt)
  4. Obtain additonal materials such as glyphs (fonts) and sprites (images)
  5. Creating a style.json file directing the client to where to find the tiles, glyphs, and sprites, as well as specifying how to style the map
  6. Uploading everything to object storage

Prerequisites

  • A working Docker installation: Our goal is to build a dockerised build pipeline, so we'll also use Docker locally for ease of setup and reproducibility.

Considerations

MBTiles vs PMTiles

There are two main formats for storing tiles:

  • MBTiles: Essentially an SQLite database containing the vector tiles. A tile server can access this database to extract and return the tiles to the client. Since we're after static hosting, the .mbtiles file would only be an intermediate result for us. We'll need to extract it into a grid layout of .pbf/.mvt files (more on that later).
  • PMTiles: All the tiles in a single flat file which can be statically hosted. Clients use the HTTP range header to access the tiles. This should work with most object storage providers, but generally does not work well with caching.

In this article, we'll be generating MBTiles first, and then extract them into .pbf/.mvt files that can be hosted statically. This is to be able to take advantage of marginally better performance and easier caching, but if you don't care about that and have a compatible client, just follow this guide till the tile generation step, change the output from MBTiles to PMTiles, upload the output file, and you're done already.

Scale

Storage requirements go up significantly with every additional zoom level. Thankfully, clients can overzoom, so generating the tiles only up to zoom level 14 will probably provide enough detail even to zoom in very far into city streets and their buildings.

Generally, the output sizes of both MBTiles, PMTiles, and extracted vector tiles up to zoom level 14 seem to be very roughly comparable to the size of the OSM extract.

Pre-generating static tiles up for the whole planet at a high zoom level might result in hundreds of millions or output files though, and most of it for tiles which none of your users will likely ever load, especially if they cover the sea. In this case, you might also want to regenerate only updated areas, based on a change set from OpenStreetMap. Optimising for this is beyond the scope of this article though.

On the other hand, generating static vector tiles up to zoom level 14 for a small but detailed country like Singapore takes only about 10 minutes on Github Actions, and finally takes up just about 20MB of storage space for about 1500 tiles.

Vector tile generators

Focusing just on tools that work from OpenStreetMap extracts (.osm.pbf files), there are still a few options:

  • Tilemaker: Simple to use tool to create an .mbtiles file directly from the OpenStreetMap extract (.osm.pbf), in a single executable.
  • Planetiler: Similar to tilemaker, but claims to be faster.
  • OpenMapTiles: The schema definition for the MBTiles format and a reference implementation for generating it. More complex to set up, using a series of tools working on a PostgreSQL database, but a Docker Compose setup and Makefile exists to make it quite usable. Rather slow, but this is fine for small extracts.

Tilemaker and Planetiler only take care of generating the .mbtiles output. They also don't support keeping your tiles up to date using OpenStreetMap diffs.

The OpenMapTiles tools comprise the whole tool chain and will help to create the other required files such as fonts as well.

We'll mainly use Planetiler here, due to it's speed and simplicity, but will provide the basic steps for OpenMapTiles as well.

Object storage providers

Most should work, but it is important that they let you set the tile file's MIME type as client libraries such as MapLibre GL JS can be picky about this. More on this later.

AWS S3, Google Cloud Storage, and Cloudflare R2 all support this. You can of course also use your own webserver such as nginx.

I'm using Cloudflare R2 in these examples, due to their low pricing and decent speed.

Client libraries

Checkout https://github.com/mapbox/awesome-vector-tiles for a list of clients for different platforms. Some interesting candidates for the web include:

Setup for MapLibre GL JS is straight-forward: Just configure it to use the style.json you created in step 5 below. For the other clients, where the documentation refers to .mvt files, you should be fine using the .pbf files extracted from the MBTiles instead (not the .osm.pbf OpenStreetMap extracts though, those are entirely different).

Running locally, step by step

1 - Obtaining the OpenStreetMap extract

You'll need to obtain an extract of the area in PBF format. There are a number of sources. Have a look here: https://wiki.openstreetmap.org/wiki/Planet.osm#Extracts

For the following steps, we'll call this file singapore.osm.pbf.

2a - Generating the MBTiles/PMTiles (Planetiler)

Create a working directory, e.g. called planetiler and place your OpenStreetMap extract inside. From this directory, run:

docker run -e JAVA_TOOL_OPTIONS="-Xmx2g" \
-v ./:/data ghcr.io/onthegomap/planetiler:latest \
--download \
--minzoom=0 \
--maxzoom=14 \
--bounds=103.49,1.105,104.59,1.535 \
--osm_path=/data/singapore.osm.pbf \
--output=/data/singapore.mbtiles

The JAVA_TOOL_OPTIONS="-Xmx2g" environment variable sets the amount of heap memory available to the process to 2GB. According to the documentation, the rule of thumb for this is about half the size of your OpenStreetMap extract, but in practice there appears to be a minimum of 2GB even for small extracts like for Singapore.

The --download option instructs Planetiler to download general source data such as global land and water outlines (~1.5GB). For subsequent runs, you can skip the --download option if you have retained the sources directory.

Be sure to specify the --bounds argument in the order West, South, East, North, otherwise Planetiler will generate millions of tiles with lesser detail for a large surrounding area.

The output format depends on the output filename, i.e. to generate PMTiles, change to --output=/data/singapore.pmtiles

Now you should have the MBTiles output file in your working directory.

2b - Generating the MBTiles (OpenMapTiles)

The OpenMapTiles repository contains a Docker Compose file and a Makefile to run all the required steps. Clone the repository, and then check out the .env file for settings. Most importantly, change the MIN_ZOOM and MAX_ZOOM settings according to your preferences. MAX_ZOOM appears to be limited to 14. Then run these steps:

make
make start-db  # Start the bundled PostgreSQL database
make import-data  # Import general data such as global land and water outlines
make import-osm ./singapore.osm.pbf  # Change to your OSM extract file
make import-wikidata
make import-sql
make generate-tiles-pg

You should now have an MBTiles file in data/tiles.mbtiles.

3 - Export MBTiles into Mapbox Vector Tiles

If you've chose to generate a PMTiles file in step 2a, you can skip this step. Otherwise, we'll need to extract the tiles from the MBTiles database into a grid layout of Mapbox Vector Tiles (.mvt or .pbf ending). The MBUtil will do it:

docker run -v ./:/data locusq/mbutil \
mb-util /data/singapore.mbtiles /data/tiles --image_format=pbf

The above command mounts the current working directory into the Docker container as /data, so adjust the path according to your MBTiles file.

Now you should have a nested set of numbered directories and files in the tiles directory. These are our static map tiles!

4 - Obtaining glyphs and sprites

WebGL can't display fonts directly, so for every font you use in your style, you'll need a Signed Distance Field glyphset in a set of .pbf files. If you're fine with using the Noto Sans family, the OpenMapTiles project has pre-generated glyphsets for the Regular, Italic and Bold variants. If you need another font, take a look at the fontnik tool.

Sprites are a map of images in a single file. If your map style uses sprites, you'll need these files:

  • sprite.json: Listing the symbols and their coordinates included in the sprite
  • sprite.png: The actual sprite
  • sprite@2x.json: The same for higher resolution
  • sprite@2x.png

If you're using one of the example styles (see next section), you can copy these files from the original.

Or you can build your own sprites from SVG files using spritezero.

5 - Create a style.json

The vector tiles we've generated so far only contain the data of which map feature is where. The style.json file specifies how it all should look like on screen. It also points to your data sources such as your vector tiles.

The easiest way to get started is Maputnik: This browser-based tool includes a number of styles as a starting point and then allows you to preview and edit all attributes.

If you've uploaded your vector tiles already, you can load them into Maputnik: Under "Data Sources" remove the existing active source, then add a new "Vector (Tile URLs)" source and edit the "Tile URL" field to match your uploaded files. "Min Zoom" and "Max Zoom" should match the tiles you've generated. This doesn't restrict the map to these zoom levels, but it will overzoom and underzoom the tiles instead of trying to access zoom levels that don't exist in your tiles. If you haven't uploaded the tiles yet, stick the the existing source first and we'll edit it later.

Then go and play with the layers, or leave everything as is. You can also use formulas to e.g. change how a layer looks like at different zoom levels.

Just an important note about fonts: When multiple font names are given for a property, they'll serve as fallbacks in case the first one cannot be loaded. They're requested as a comma-separated URL, which won't work with our static hosting setup. But we also don't need such fallbacks, because we know exactly which fonts we'll be hosting. So make sure to only specify a single font per property. The examples usually use fallback fonts and this will just make any font fail to load. As noted above, it's easiest to stick to the "Noto Sans" family.

When you're done, click "Save" to download the file. Then we'll still need to make a few manual changes to the file:

For your sources, consider setting the bounds and attribution attributes, e.g.

"sources": {
  "my-static-tiles": {
    "type": "vector",
    "tiles": [
      "https://your-tile-host.example/tiles/{z}/{x}/{y}.pbf"
    ],
    "maxzoom": 14,
    "bounds": [103.49,1.105,104.59,1.535],
    "attribution": "© <a href=\"https://carto.com/about-carto/\" target=\"_blank\" rel=\"noopener\">CARTO</a>, © <a href=\"https://www.openmaptiles.org/\" target=\"_blank\">OpenMapTiles</a>, © <a href=\"http://www.openstreetmap.org/copyright\" target=\"_blank\">OpenStreetMap</a> contributors"
  }
},
"glyphs": "https://your-tile-host.example/fonts/{fontstack}/{range}.pbf",
"sprite": "https://your-tile-host.example/sprite",

The bounds instructs the client not to try to load tiles outside of these boundaries and avoids request errors for missing tiles.

The example styles are usually copyrighted and require you to provide attribution. Using OpenStreetMap data also requires attribution. In this example, I've used Carto's excellent version of OpenMapTiles' Positron style, so I need to provide attribution for them both. Many clients will use this field to automatically include an attribution note on the map.

This is also where you can adapt the paths to your tiles, glyphs and sprites. And double check that every text-font property only contains a single item!

If you've generated PMTiles, simply point your tiles path to that file, no x-y-z variables required.

6 - Setup static hosting

In this example, we'll use the following directory structure for hosting our tiles and supporting files.

/style.json
/tiles/{z}/{x}/{y}.pbf  # The tiles
/fonts/  # Containing a folder of .pbf files for each font
/sprite/  # Containing the .png and .json files for the sprite

The .pbf files containing tiles and glyphs are Protocol Buffers, and in addition, the tiles are already gzip encoded. Unfortunately, most tools don't automatically recognise them as such. Clients expect the Content-Type: application/x-protobuf and Content-Encoding: gzip headers when retrieving the .pbf files. 

A simple nginx config to achieve this (e.g. for hosting the tiles locally) could look like this:

server {
    listen 80;

    types {
        application/x-protobuf pbf;
    }
    
    location / {
        root /usr/share/nginx/html;
        add_header Access-Control-Allow-Origin "*" always;
    }
    
    location /tiles/ {
        root /usr/share/nginx/html;
        add_header Access-Control-Allow-Origin "*" always;
        add_header Content-Type "application/x-protobuf";
        add_header Content-Encoding "gzip";
    }
    
    location /fonts/ {
        root /usr/share/nginx/html;
        add_header Access-Control-Allow-Origin "*" always;
        add_header Content-Type "application/x-protobuf";
    }
}

For object storage such as AWS S3 or Cloudflare R2, you'll have to specify these headers when uploading the files, e.g. with rclone:

docker run -v ./tiles:/tiles \
-e RCLONE_CONFIG_R2_TYPE="s3" \
-e RCLONE_CONFIG_R2_PROVIDER="Cloudflare" \
-e RCLONE_CONFIG_R2_ACCESS_KEY_ID="$RCLONE_CONFIG_R2_ACCESS_KEY_ID" \
-e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$RCLONE_CONFIG_R2_SECRET_ACCESS_KEY" \
-e RCLONE_CONFIG_R2_ENDPOINT="$RCLONE_CONFIG_R2_ENDPOINT" \
-e RCLONE_CONFIG_R2_ACL="public-read" \
rclone/rclone:v1.69-stable copy \
--header-upload "Content-Type: application/x-protobuf" \
--header-upload "Content-Encoding: gzip" \
--ignore-size \
/tiles r2:{bucket-name}/tiles/

The same applies for uploading the fonts, sans the Content-Encoding header.

Running on GitHub Actions

Using Planetiler

This workflow fully regenerates all tiles and uploads them to R2 on the 5th of every month. It takes about 3 minutes to generate the tiles for Singapore and then another 7 minutes to upload them to R2.

Since glyphs, sprites, and the style.json don't need to be updated regularly, you can just upload them manually once.

name: Build map tiles
on:
  schedule:
    - cron: "37 4 5 * *"
  workflow_dispatch:
env:
  RCLONE_CONFIG_R2_TYPE: s3
  RCLONE_CONFIG_R2_PROVIDER: Cloudflare
  RCLONE_CONFIG_R2_ENDPOINT: https://{bucket-endpoint}.r2.cloudflarestorage.com
  RCLONE_CONFIG_R2_ACL: public-read
jobs:
  Build-Map-Tiles:
    runs-on: 'ubuntu-latest'
    timeout-minutes: 40
    steps:
      - name: Create tiles directory
        run: mkdir ./tiles
      - name: Download extract
        working-directory: ./tiles
        run: wget https://download.openstreetmap.fr/extracts/asia/singapore.osm.pbf
      - name: Generate tiles
        working-directory: ./tiles
        run: >-
          docker run -e JAVA_TOOL_OPTIONS="-Xmx2g"
          -v ./:/data ghcr.io/onthegomap/planetiler:latest
          --download
          --minzoom=0
          --maxzoom=14
          --bounds=103.49,1.105,104.59,1.535
          --osm_path=/data/singapore.osm.pbf
          --output=/data/tiles.mbtiles
      - name: Extract tiles
        working-directory: ./tiles
        run: docker run -v ./:/data locusq/mbutil mb-util /data/tiles.mbtiles /data/out --image_format=pbf
      - name: Upload tiles
        env:
          RCLONE_CONFIG_R2_ACCESS_KEY_ID: ${{ secrets.MAP_TILES_R2_ACCESS_KEY_ID }}
          RCLONE_CONFIG_R2_SECRET_ACCESS_KEY: ${{ secrets.MAP_TILES_R2_SECRET_ACCESS_KEY }}
        working-directory: ./tiles
        run: >-
          docker run -v ./:/tiles 
          -e RCLONE_CONFIG_R2_TYPE="$RCLONE_CONFIG_R2_TYPE"
          -e RCLONE_CONFIG_R2_PROVIDER="$RCLONE_CONFIG_R2_PROVIDER"
          -e RCLONE_CONFIG_R2_ACCESS_KEY_ID="$RCLONE_CONFIG_R2_ACCESS_KEY_ID"
          -e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$RCLONE_CONFIG_R2_SECRET_ACCESS_KEY"
          -e RCLONE_CONFIG_R2_ENDPOINT="$RCLONE_CONFIG_R2_ENDPOINT"
          -e RCLONE_CONFIG_R2_ACL="$RCLONE_CONFIG_R2_ACL"
          rclone/rclone:v1.69-stable copy
          --header-upload "Content-Type: application/x-protobuf" --header-upload "Content-Encoding: gzip" --ignore-size
          /tiles/out r2:map-tiles/tiles/

Using OpenMapTiles

This workflow covers the whole flow, including downloading the Noto Sans fonts and sprites (which is only required for the first run) and takes about 20 minutes on a free runner, about half of this for generating the files, and the other half for the rclone upload to R2. All that's missing is to manually upload the style.json (also only required once).

name: Build map tiles
on:
  schedule:
    - cron: "37 4 5 * *"
  workflow_dispatch:
env:
  RCLONE_CONFIG_R2_TYPE: s3
  RCLONE_CONFIG_R2_PROVIDER: Cloudflare
  RCLONE_CONFIG_R2_ENDPOINT: https://{bucket-endpoint}.r2.cloudflarestorage.com
  RCLONE_CONFIG_R2_ACL: public-read
jobs:
  Build-Map-Tiles:
    runs-on: 'ubuntu-latest'
    timeout-minutes: 40
    steps:
      - name: Checkout openmaptiles
        uses: actions/checkout@v4
        with:
          repository: openmaptiles/openmaptiles
          ref: v3.15
          path: openmaptiles/
      - name: Set max zoom level
        working-directory: ./openmaptiles
        run: sed -i 's|MAX_ZOOM=7|MAX_ZOOM=14|' .env
      - name: Run make
        working-directory: ./openmaptiles
        run: make
      - name: Start DB
        working-directory: ./openmaptiles
        run: make start-db
      - name: Import data
        working-directory: ./openmaptiles
        run: make import-data
      - name: Download extract
        working-directory: ./openmaptiles
        run: make download-osmfr area=asia/singapore
      - name: Import OSM
        working-directory: ./openmaptiles
        run:   
      - name: Import Wikidata
        working-directory: ./openmaptiles
        run: make import-wikidata
      - name: Import SQL
        working-directory: ./openmaptiles
        run: make import-sql
      - name: Download fonts
        working-directory: ./openmaptiles
        run: make download-fonts
      - name: Build sprite
        working-directory: ./openmaptiles
        run: make build-sprite
      - name: Generate tiles
        working-directory: ./openmaptiles
        run: make generate-tiles-pg
      - name: Process tiles
        working-directory: ./openmaptiles
        run: mkdir tiles && docker run -v ./data:/data -v ./tiles:/tiles locusq/mbutil mb-util /data/tiles.mbtiles /tiles/out --image_format=pbf
      - name: Upload tiles
        env:
          RCLONE_CONFIG_R2_ACCESS_KEY_ID: ${{ secrets.MAP_TILES_R2_ACCESS_KEY_ID }}
          RCLONE_CONFIG_R2_SECRET_ACCESS_KEY: ${{ secrets.MAP_TILES_R2_SECRET_ACCESS_KEY }}
        working-directory: ./openmaptiles
        run: >-
          docker run -v ./tiles:/tiles 
          -e RCLONE_CONFIG_R2_TYPE="$RCLONE_CONFIG_R2_TYPE"
          -e RCLONE_CONFIG_R2_PROVIDER="$RCLONE_CONFIG_R2_PROVIDER"
          -e RCLONE_CONFIG_R2_ACCESS_KEY_ID="$RCLONE_CONFIG_R2_ACCESS_KEY_ID"
          -e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$RCLONE_CONFIG_R2_SECRET_ACCESS_KEY"
          -e RCLONE_CONFIG_R2_ENDPOINT="$RCLONE_CONFIG_R2_ENDPOINT"
          -e RCLONE_CONFIG_R2_ACL="$RCLONE_CONFIG_R2_ACL"
          rclone/rclone:v1.69-stable copy
          --header-upload "Content-Type: application/x-protobuf" --header-upload "Content-Encoding: gzip" --ignore-size
          /tiles/out r2:map-tiles/tiles/
      - name: Upload fonts
        env:
          RCLONE_CONFIG_R2_ACCESS_KEY_ID: ${{ secrets.MAP_TILES_R2_ACCESS_KEY_ID }}
          RCLONE_CONFIG_R2_SECRET_ACCESS_KEY: ${{ secrets.MAP_TILES_R2_SECRET_ACCESS_KEY }}
        working-directory: ./openmaptiles
        run: >-
          docker run -v ./data/fonts:/fonts
          -e RCLONE_CONFIG_R2_TYPE="$RCLONE_CONFIG_R2_TYPE"
          -e RCLONE_CONFIG_R2_PROVIDER="$RCLONE_CONFIG_R2_PROVIDER"
          -e RCLONE_CONFIG_R2_ACCESS_KEY_ID="$RCLONE_CONFIG_R2_ACCESS_KEY_ID"
          -e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$RCLONE_CONFIG_R2_SECRET_ACCESS_KEY"
          -e RCLONE_CONFIG_R2_ENDPOINT="$RCLONE_CONFIG_R2_ENDPOINT"
          -e RCLONE_CONFIG_R2_ACL="$RCLONE_CONFIG_R2_ACL"
          rclone/rclone:v1.69-stable copy
          --header-upload "Content-Type: application/x-protobuf" --ignore-size
          /fonts r2:map-tiles/fonts/
      - name: Upload sprite
        env:
          RCLONE_CONFIG_R2_ACCESS_KEY_ID: ${{ secrets.MAP_TILES_R2_ACCESS_KEY_ID }}
          RCLONE_CONFIG_R2_SECRET_ACCESS_KEY: ${{ secrets.MAP_TILES_R2_SECRET_ACCESS_KEY }}
        working-directory: ./openmaptiles
        run: >-
          docker run -v ./build/style:/style
          -e RCLONE_CONFIG_R2_TYPE="$RCLONE_CONFIG_R2_TYPE"
          -e RCLONE_CONFIG_R2_PROVIDER="$RCLONE_CONFIG_R2_PROVIDER"
          -e RCLONE_CONFIG_R2_ACCESS_KEY_ID="$RCLONE_CONFIG_R2_ACCESS_KEY_ID"
          -e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$RCLONE_CONFIG_R2_SECRET_ACCESS_KEY"
          -e RCLONE_CONFIG_R2_ENDPOINT="$RCLONE_CONFIG_R2_ENDPOINT"
          -e RCLONE_CONFIG_R2_ACL="$RCLONE_CONFIG_R2_ACL"
          rclone/rclone:v1.69-stable copy
          --exclude=style.json
          /style r2:map-tiles/sprite/