Using the Overpass API to read OSM data, parsing the data with Rust, and then drawing the map onto HTML5 canvas.
I’m working on a revamp of an old project of mine called Line Buddy (github). It used a now-deprecated API library called themeparks (github) and A-Frame to visually represent the wait times in the Disney World theme parks in 3D.
The original project used OpenStreetMap screenshots as the base, with columns representing the wait times. (They’re all zero now since this version of the API no longer works.)
My plan is to use OpenStreetMap data to create a simplified version of the map. Eventually, I’ll create the map in 3D. As a proof of concept, however, I’m going to draw to an HTML5 canvas first. I want to make sure that I’m able to get the data that I need, process it, and use it to create my own maps.
Table of Contents
- Setting up the project
- Creating a new project with wasm-pack
- Building and testing WASM code
- Getting map coordinates
- Calling the Overpass API
- Writing the Rust code
- Drawing on the canvas
- Processing ways and relations
- Drawing all of the other map areas
- The finished map
Setting up the project
I’m not going to use any sort of JavaScript library for this first experiment. I set up a basic HTML page inside of a directory, and I’m adding CSS and JavaScript directly to this page. A very simple setup because all I need the JavaScript to do is call the WASM code and draw to a canvas.
DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>OSM Mapstitle>
<style>
body {
margin: 0;
padding: 0;
height: 100%;
width: 100%;
}
style>
head>
<body>
<canvas id="canvas">canvas>
<script type="module">script>
body>
html>Most of the heavy lifting for the OSM data processing will be taken care of by Rust. While I haven’t performed any benchmarks about whether Rust or JavaScript processing is faster, I decided to use Rust and WebAssembly because there will be a lot of coordinates to process, and Rust generally performs faster at large amounts of data processing.
Creating a new project with wasm-pack
I decided to use wasm-pack to create a Rust library that will then be used by my client-side JavaScript. While I already had wasm-pack installed, I re-ran the installer because my installed version of wasm-pack was extremely out of date.
Creating a new wasm-pack project is easy. Instead of my normal workflow of using cargo new process-maps to create my new Rust project, I used wasm-pack new process-maps. This creates a new Rust project that is optimized for creating WASM files.
I created this project inside the same directory as my HTML file. This is because I want to be able to easily call the WASM code from my HTML file.
Building and testing WASM code
When I open up the lib.rs file generated by the wasm-pack tool, I see that it already contains some minimal code that will display an alert whenever the greet() function is called from JavaScript. There’s a line-by-line explanation of this file at https://rustwasm.github.io/docs/wasm-pack/tutorials/npm-browser-packages/template-deep-dive/src-lib-rs.html.
mod utils;
use wasm_bindgen::prelude::*;
#[wasm_bindgen]
extern "C" {
fn alert(s: &str);
}
#[wasm_bindgen]
pub fn greet() {
alert("Hello, process-maps!");
}I am going to compile this code and import it into my HTML file.
wasm-pack build --target web --devThis creates a development build of my Rust code. Running the command without --dev will compile a production release, which is slower but optimized. I’m targeting “web” since I’m not using any sort of bundler or webpack in my project.
Under the canvas tag in my HTML file, I’m going to import the WASM code and call the greet function.
<script type="module">
import init, { greet } from './process-maps/pkg/process_maps.js';
async function run() {
await init();
greet();
}
run();
script>When I open my HTML file in a browser, I see an alert that says “Hello, process-maps!” This means that my WASM code is working and I can start writing the code to process the OSM data.
Getting map coordinates
Before I can write any Rust code, I need to get the coordinates for a location. In this case, I want to get the coordinates for the different Disney World parks.
Because I want to pick a very specific area of the map, I will navigate to openstreetmap.org and search for the locations I want to map. Then, I’ll click the “Export” button. In the Export sidebar, I’ll click “Manually select a different area”.
Once I have the area cropped, I’m going to add the coordinates to my code. Since I’m going to be creating maps of all the Disney parks, I’m using query parameters to differentiate between which park map to draw.
const queryString = window.location.search;
const searchParams = new URLSearchParams(queryString);
let park = searchParams.get('park');
let n, w, s, e;
switch (park) {
case 'epcot':
n = 28.3768;
w = -81.5553;
s = 28.3661;
e = -81.5425;
break;
case 'hollywood_studios':
case 'hs':
n = 28.3625;
w = -81.5641;
s = 28.3523;
e = -81.5561;
break;
case 'animal_kingdom':
case 'ak':
n = 28.3692;
w = -81.5984;
s = 28.3524;
e = -81.5831;
break;
case 'magic_kingdom':
case 'mk':
default:
n = 28.42266;
w = -81.58586;
s = 28.41604;
e = -81.57600;
break;
}Calling the Overpass API
OpenStreetMap’s public API is called the Overpass API. It has an interesting query language that I’m not totally sure I understand, but I was able to create some queries that return the data I need.
Below, I have created query strings for all of the map items I need: walkways, gardens, water, and buildings. Below that, I’m fetching the data for walkways.
// The bounding box is described with the south, west, north, and east coordinates, in that order.
let bbox = `${s},${w},${n},${e}`;
// Queries for each map type.
let buildings = `way[building][!name];foreach{(._;>;);out;}`;
let named_buildings = `way[building][name];foreach{(._;>;);out;}`;
let walkways = `way[highway];foreach{(._;>;);out;}`;
let trees = `node[natural=tree];foreach{(._;>;);out;}`;
let gardens = `(way[leisure=garden];way[landuse=forest];way[landuse=meadow];);foreach{(._;>;);out;}`;
let water = `relation[natural=water];foreach{(._;>;);out;}`;
// Sets the timeout and bounding box.
let query = `[timeout:90][bbox:${bbox}];`;
let url = `https://overpass-api.de/api/interpreter?data=${query}`;
function getWalkways(url) {
fetch(`${url}${walkways};out;`).then(response => {
return response.text();
}).then(data => {
console.log(data);
});
}The response in the console should look somewhat like this, a long XML document with coordinates.
I’m going to send these coordinates to my Rust code, which will parse the XML file and return coordinates that I’ll then be able to map to the canvas.
Writing the Rust code
To parse the XML file, I’m using a crate called osm-xml. It abstracts away the hard parts of parsing the OSM data. I tried doing it without this crate and trust me, it’s very difficult.
I’m adding osm-xml and js-sys to my Cargo.toml. The crate js-sys provides interop data structures for JavaScript.
[dependencies]
wasm-bindgen = "0.2.84"
console_error_panic_hook = { version = "0.1.7", optional = true }
osm-xml = "0.6.2"
js-sys = "0.3.67"OSM data can be thought of as three different types. There’s nodes, which are discrete coordinates. The query for trees searches for nodes. Ways are polygons that are created from nodes, so processing a way requires processing the information for the way, and then the nodes. Buildings, walkways, and gardens are ways. Finally, there are relations, which are made of multiple ways. Water is a relation because there can be islands within the water. Any time there might need to be holes cut within a larger polygon, it will be a relation.
Since nodes are the easiest to parse, I’m starting with processing the nodes first, then the ways, then relations.
First, I’m going to turn my “greet” function into a log function. This way, I’ll be able to log output to the JS console.
#[wasm_bindgen]
extern "C" {
#[wasm_bindgen(js_namespace = console)]
fn log(s: &str);
}Next, I’m passing the string provided by the API (in the fetch call I’m returning response.text()) to my WASM-bound function.
#[wasm_bindgen]
pub fn process_nodes(text: String) -> js_sys::Array {
// Parses the string into the OSM data structure.
let doc = osm::OSM::parse(text.as_bytes()).unwrap();
// Bounds are the min and max latitudes and longitudes of the map.
let bounds = doc.bounds.unwrap();
// New JS array that will be returned.
let arr = js_sys::Array::new();
// Iterate over the nodes found in the document.
// Each node should be the location of a tree.
for (_id, node) in doc.nodes.iter() {
// The nodes will be processed and pushed to the JS array,
// but for now, just logging out the data.
log(&format!("{} {}"), node.lat, node.lon);
}
arr
}Now I have to call this function from JavaScript. I’ll create a getTrees function that fetches data from the trees endpoint, then call the WASM process_nodes function.
// First, update the import at the top of the script.
import init, {process_nodes} from './process-maps/pkg/process_maps.js';
// Add getTrees to the bottom of the script.
function getTrees(url) {
fetch(`${url}${trees};out;`).then(response => {
return response.text();
}).then(data => {
process_nodes(data, width, height);
});
}
getTrees(url);After rebuilding the WASM code and refreshing the page, I see lists of coordinates in my developer console.
I can’t just use raw coordinates to draw to a canvas, however. I need to convert the geographical coordinates to xy coordinates that will map to the size of my canvas.
First, I need a function that will take a single coordinate value and map it to the range of 0 to the width or height of the canvas.
// start1 and stop1 are the min and max map bounds that were unwrapped earlier.
// start2 and stop2 are the min and max values of the canvas.
fn map_points(value: f64, start1: f64, stop1: f64, start2: f64, stop2: f64) -> f64 {
((value - start1) / (stop1 - start1) * (stop2 - start2) + start2).floor()
}Next, I need to create the x and y coordinates and wrap them in an array for export to my JavaScript.
fn process_points(node: &osm::Node, bounds: &osm::Bounds, width: f64, height: f64) -> js_sys::Array {
let y = map_points(node.lat, bounds.minlat, bounds.maxlat, 0.0, width);
let x = map_points(node.lon, bounds.minlon, bounds.maxlon, 0.0, height);
let point = js_sys::Array::new();
// Numbers being sent to JavaScript must be transformed into a JsValue.
point.push(&JsValue::from_f64(x));
point.push(&JsValue::from_f64(y));
point
}Finally, I can add a call to process_points from my process_nodes function.
#[wasm_bindgen]
pub fn process_nodes(text: String, width: f64, height: f64) -> js_sys::Array {
let doc = osm::OSM::parse(text.as_bytes()).unwrap();
let bounds = doc.bounds.unwrap();
let arr = js_sys::Array::new();
for (_id, node) in doc.nodes.iter() {
arr.push(&process_points(node, &bounds, width, height));
}
arr
}Now, calling this function from my JavaScript file should return an array of points. Since I’m actually returning an array, I can stop calling log from the Rust script, and I can call console.log from JavaScript.
function getTrees(url) {
fetch(`${url}${trees};out;`).then(response => {
return response.text();
}).then(data => {
let p = process_nodes(data, width, height);
console.log(p);
});
}
getTrees(url);This returns a large array of arrays with length 2. Each point corresponds to a point on the canvas.
Drawing on the canvas
Now that there are point coordinates, they can be drawn to the canvas itself. First, I need to prepare the canvas for drawing.
let canvas = document.getElementById('canvas');
let ratio = (Math.abs(w) - Math.abs(e)) / (n - s);
let width = 1800;
let height = width * ratio;
canvas.width = height;
canvas.height = width;
let ctx = canvas.getContext('2d');
ctx.translate(0, width);
ctx.scale(1, -1);
ctx.fillStyle = '#ffffff';
ctx.fillRect(0, 0, height, width);I had to do some rotating and flipping of the canvas to get the map to draw in the correct direction. Perhaps this is a problem with my coordinate mapping, or some other issue, but this seems to fix it. Next, I have to get the processed tree coordinates, and once those are available, draw the trees to the canvas.
const tree_data = getTrees(url); // Returns the processed coordinates.
// Waits for all functions to resolve.
// Right now, it's just the getTrees function.
Promise.all([tree_data]).then(values => {
const [trees] = values;
for (let tree of trees) {
ctx.beginPath();
ctx.arc(tree[0], tree[1], 3, 0, 2 * Math.PI);
ctx.fillStyle = 'rgb(40,107,83)';
ctx.fill();
ctx.closePath();
}
});
// Updated this function to return the fetched and processed data.
function getTrees(url) {
return fetch(`${url}${trees};out;`).then(response => {
return response.text();
}).then(data => {
return process_nodes(data, width, height);
});
}All the trees should now be drawn to the canvas.
Processing ways and relations
To draw any other items to the map, I need to add the ability to process OpenStreetMap ways and relations to my Rust code. This is my initial code to extract ways from the OSM data. It’s nearly identical to the process_nodes function, but I’ll need to add another processing step.
#[wasm_bindgen]
pub fn process_ways(text: String, width: f64, height: f64) -> js_sys::Array {
let doc = osm::OSM::parse(text.as_bytes()).unwrap();
let bounds = doc.bounds.unwrap();
let arr = js_sys::Array::new();
for (_id, way) in doc.ways.iter() {
log(&format!("{:?}", way));
}
arr
}Then, I import this function into my JavaScript code and call it.
import init, {process_nodes, process_ways} from './process-maps/pkg/process_maps.js';
const walkway_data = getWalkways(url);
function getWalkways(url) {
return fetch(`${url}${walkways};out;`).then(response => {
return response.text();
}).then(data => {
return process_ways(data, width, height);
});
}Logging this out will return a “Way” object, which contains multiple nodes.
Now, I need to write an additional function that will process each of the nodes in the way and create canvas coordinates.
fn process_coords(doc: &osm::OSM, way: &osm::Way, bounds: &osm::Bounds, width: f64, height: f64) -> js_sys::Array {
// Create an empty array to return the coordinates.
let coords = js_sys::Array::new();
// Iterate over each of the nodes in the way.
for node in way.nodes.iter() {
// Finds the node by its ID number.
let n = &doc.resolve_reference(&node);
// Matches the node by reference and then generates coordinates.
match n {
osm::Reference::Node(node) => {
let point = process_points(node, &bounds, width, height);
coords.push(&point);
},
_ => {}
}
}
coords
}I need to call this function from process_ways.
#[wasm_bindgen]
pub fn process_ways(text: String, width: f64, height: f64) -> js_sys::Array {
let doc = osm::OSM::parse(text.as_bytes()).unwrap();
let bounds = doc.bounds.unwrap();
let arr = js_sys::Array::new();
for (_id, way) in doc.ways.iter() {
arr.push(&process_coords(&doc, way, &bounds, width, height));
}
arr
}Logging the result out from JavaScript will return an array of all the walkways, broken down into coordinate pairs.
Processing the relations (waterways, in the case of this map) requires a further breakdown step. Relations need to be mapped to ways, which are then mapped to nodes. When I process the ways, I’m only going to push the coordinates for the “outer” way to my array. The “inner” ways are cutout areas of the larger polygon. For my purposes, I’m going to draw the water first, then the land and everything else, so I really don’t need to worry about the inner polygons. This cuts down on extra processing and drawing time.
#[wasm_bindgen]
pub fn process_relations(text: String, width: f64, height: f64) -> js_sys::Array {
let doc = osm::OSM::parse(text.as_bytes()).unwrap();
let bounds = doc.bounds.unwrap();
let arr = js_sys::Array::new();
for (_id, relation) in doc.relations.iter() {
for member in relation.members.iter() {
match member {
// Matching ways within the relation.
osm::Member::Way(way, t) => {
let w = &doc.resolve_reference(&way);
match w {
osm::Reference::Way(way) => {
// Only processing coordinates for the "outer" way.
if t == "outer" {
arr.push(&process_coords(&doc, way, &bounds, width, height));
}
},
_ => {}
}
},
_ => {}
}
}
}
arr
}Drawing all of the other map areas
Since the rest of the map is constructed with filled polygons, I created a function to draw each polygon for the various map types. Each type of map item will have a different color, and walkways will only have a stroke and no fill.
Below is all the code that fetches the OSM data for each landmark, calls the WASM functions to process the data, then draws the coordinates to the canvas. I’ve left out the Overpass API code, which can be referenced above at calling the Overpass API.
const url = `https://overpass-api.de/api/interpreter?data=${query}`;
const water_data = getWater(url);
const garden_data = getGardens(url);
const walkway_data = getWalkways(url);
const tree_data = getTrees(url);
const building_data = getBuildings(url);
const nbuilding_data = getNamedBuildings(url);
Promise.all([water_data, garden_data, walkway_data, tree_data, building_data, nbuilding_data]).then(values => {
const [water, gardens, walkways, trees, buildings, named_buildings] = values;
drawPolygons(water, 'rgb(83,156,156)', null);
drawPolygons(gardens, 'rgb(136,172,140)', null);
drawPolygons(walkways, null, 'rgb(0,0,0)');
for (let tree of trees) {
ctx.beginPath();
ctx.arc(tree[0], tree[1], 3, 0, 2 * Math.PI);
ctx.fillStyle = 'rgb(40,107,83)';
ctx.fill();
ctx.closePath();
}
drawPolygons(buildings, 'rgb(98,90,87)', null);
drawPolygons(named_buildings, 'rgb(220,177,102)', null);
});
function drawPolygons(p, fill, stroke) {
for (let polygon of p) {
ctx.beginPath();
for (let point of polygon) {
ctx.lineTo(point[0], point[1]);
}
if (fill) {
ctx.fillStyle = fill;
ctx.fill();
}
if (stroke) {
ctx.strokeStyle = stroke;
ctx.stroke();
}
ctx.closePath();
}
}
function getWater(url) {
return fetch(`${url}${water};out;`).then(response => {
return response.text();
}).then(data => {
return process_relations(data, width, height);
});
}
function getWalkways(url) {
return fetch(`${url}${walkways};out;`).then(response => {
return response.text();
}).then(data => {
return process_ways(data, width, height);
});
}
function getBuildings(url) {
return fetch(`${url}${buildings};out;`).then(response => {
return response.text();
}).then(data => {
return process_ways(data, width, height);
});
}
function getNamedBuildings(url) {
return fetch(`${url}${named_buildings};out;`).then(response => {
return response.text();
}).then(data => {
return process_ways(data, width, height);
});
}
function getGardens(url) {
return fetch(`${url}${gardens};out;`).then(response => {
return response.text();
}).then(data => {
return process_ways(data, width, height);
});
}
function getTrees(url) {
return fetch(`${url}${trees};out;`).then(response => {
return response.text();
}).then(data => {
return process_nodes(data, width, height);
});
}The finished map
I’ve benchmarked the map’s loading time, and it takes about 5.2 seconds on average when the Rust code is compiled in development mode, and about 2.3 seconds in production mode. In an earlier iteration I had each map endpoint individually queried, processed, and drawn before querying the next endpoint, which was much slower. Now, the map is at the mercy of the slowest query (around 2.1 seconds). A way to save more time would be to save the coordinates to a file instead of querying the API, but any changes wouldn’t be reflected in the map.
Here’s a finished screenshot of the Magic Kingdom’s map.
This is part one of a much larger project, so while it feels a bit janky now, everything will be cleaned up in the future. The main goal here was to prove that it is possible to use Rust and WASM to process these coordinates to make my own maps.
