Analysing OSM data for planning purposes – not so easy

It’s just so easy to add new features and edit those existing on openstreetmap with it’s impressive number of tools including iD and JOSM. Little know though is the vast array of little tools which have been created for extracting and working with that wonderful open geo data.

This last week has been eye opening for me once again, I have been working with a small group of assistant physical planners on how we get hold of an extract of OSM for Lesotho, how do we load this into our chosen GIS tool (postGIS) and finally what sorts of exploratory analysis is possible.

Thanks to the lovely folks at geofabrik we have a daily Lesotho OSM extract. It takes the form of an osm.pbf file.

It was a week of new experiences for all the APPs, they had for one never touched the command line before. Something you really need to know if you want to use osm2pgsql to get your pbf file into your database.

Each day they will download the lesotho-latest.osm.pbf file from geofabrik and open up the command line to run the following:

osm2pgsql -c -d lesotho0902 -U colinbroderick -H localhost --hstore  --slim --number-processes 10 --extra-attributes -C 10000 1602-lesotho-latest.osm.pbf

This creates at least 3 database tables one for each of the geometry types present in OSM:

  • planet_osm_point
  • planet_osm_line
  • planet_osm_polygon

Screen Shot 2016-02-18 at 10.48.29

We are now able to easily perform some high level spatial analysis using the data in each of these tables.

Have you used the command line before? It’s not so easy to get started with right? After a rocky start all our guys are flying along with it now, it’s no longer scary! They are updating the database everyday!


We chose to take on the challenge of training the APP’s to use PostGIS for their analysis as it we felt it was the most performant for them to work between periods of online connectivity.

Watch out for some details of the spatial analysis that will be falling out of these five newly trained spatial analysts!

Find out more on how you can get your own Lesotho database on our analysis wiki

#MapLesotho – where is it going?

#MapLesotho has rocketed forwards in the last ten weeks. Thanks to the @Mapillary challenge the number of nodes  in the map has double in that time from 3.6 million to 7 million. Now there are regularly between 6-12 mappers engaging in mapping the mountain kingdom every day.  Lesotho, if not the most mapped, is one of the most mapped African countries.

Rise in OSM Nodes June-Sept 2015

Rise in OSM Nodes June-Sept 2015

Now it is timely to look ahead at how the big push will be made to finish the basemap and move on to later phases.  In February 2015 Dave Corley conducted a consultation with the Lesotho OSM stakeholders that sketched out the general principles of layering in different data sources, tracing their features and creating stronger and stronger aggregations of locally mapped features until openstreetmap for Lesotho fills up, to perhaps be the most completely mapped country in the world.

From the present time #hotosm task 1169 will need to be completed. When this is done all mappers will be asked to validate all the completed tiles within the large Rural 1, Urban 1, Rural 2 and Urban 2 tasks.

Qacha's Nek HOTOSM task

Qacha’s Nek HOTOSM task

Simultaneous to this stage older urban tasks completed for Mafeteng, Teyataneng and Qachas’ Nek from 2014 need a lot of the mapping re-rectified which will be completed centrally.

The next objective is to secure building level mapping for the rural areas, since the districts now have teams of OSM users and the complexity of this work the task will be set up on a district basis. Districts in the west will have an advantage as they were partly mapped for all their buildings in the other urban tasks. Of course districts are all different sizes too, so the speed of this task will vary greatly from place to place and a coterie of mappers from the #hotosm crowdsource will be needed if some are going very slowly.  At this stage it could be said that the base-map is both complete and validated.

While the features are still quite basic in this base-map there is an intention to run basic error detection tools, like OSM Inspector devised by Geofabrick. This will detect indications of mistakes made by mappers that were not ironed out by validation. For examples streams flowing in the wrong direction, or abrupt ends to highways, buildings that are conjoined at a corner, and so on.

The very next step is to make the base map as useable as possible to wider numbers of people. Using out-of-copyright raster maps from the Bodleian Library and the University of Texas as many additional details as can be gleaned, like heights and village/settlement place-names.

Out of copyright maps of Lesotho: source University of Texas

Out of copyright maps of Lesotho: source University of Texas

These will also be done on a district basis. Each area marked out as a residential in the previous task will be scanned for an established place name.

The next task will be bulk uploading, pending their release free of copyright, the Land Administration Authority (LAA) boundary files for District and Community Council organisation.

Lesotho's Community Councils: source LAA

Lesotho’s Community Councils: source LAA

The LAA will be asked to release additional data too, some of this may already be mapped, however, and this will require mappers to reconcile what may be points as orthodox drawn features in openstreetmap.  For example footbridges exist in their data as a point (pin), whereas in OSM it is customary to present them as a section on a footpath.

The phase which follows is aimed at using (perhaps in parallel) OSM methodologies like Mapillary and/or Walking Papers surveys to add as much as possible about building uses, schools, shops, medical centres, police stations, street lights, footpaths and so forth can all be drawn on the map at this stage.

Walking Papers Print-out for ~MapLesotho

Walking Papers Print-out for ~MapLesotho

At this stage the real value to apps developers emerges as they use the tags that are created on each building.

Of course the Map of Lesotho, just like the map of any country, will never be entirely correct. It will require a community of custodians who regularly check, improve, nuance and edit openstreetmap and keep track of items like new buildings and demolished buildings. Having said that the map will be in a highly useable state at this point.

At this point tools which allow “maintenance editing” to take place will be explored. One excellent example of just such a tool is the one developed by Jóhannes Jenson. This tool allows a view of settlements, and allows them to be rated and ultimately flagged up and edited.

A summary of all these phases, with some time-lines is set out here:

Next Steps for #MapLesotho

Next Steps for #MapLesotho

Of course there is no reserve on good ideas. The #MapLesotho community is very interested in hearing from anyone who has a suggestion. These ideas can be posted on twitter using the hashtag #MapLesotho, or to the OSM Lesotho facebook page.

Take a spin around Lesotho with Mapillary

downloadMapillary is a mobile and tablet application which enables picture capture and sharing, as well as geo-tagging. It allows pictures to be assembled to form street views and panoramas. However, unlike the universally known google street view the picture data is crowdsourced, and so is free of copyright. The map is downloaded free from the play store, and app store.

Mapillary Icon

Mapillary Icon

 Mapillary is available to openstreetmap users in ID editor as a layer, and now lately in JOSM as a plugin. This gives a mapper a ground level view of a feature, additional to the top-down view which can limit perspective, and the likelyhood of correct interpretation and tagging. It gives the chance also to overcome the blurring and lack of rectification of some of the satellite tiles for Lesotho.
In February 2014 the Fingal Trainers in Lesotho tested Mapillary over a period of 10 days to see if it would be useful as a reference that would speed up mapping, and improve the accuracy of mapping. Thousands of images were captured by Dave Corley as the team travelled around urban Maseru and to the airport. A longer drive south to Ha’Mats’ebo on the border between Maseru and Mafeteng districts tested the usefulness of the application in rural areas. This drive took the opportunity to head east, into the generally flat terrain of Ha Phohleli Makoaleng, and then north to Mahlabateng.
Test route

Rural test route

A good example of a feature that cannot be interpreted correctly in Urban Lesotho is an overhead gantry. These gantries serve no traffic management purpose in Lesotho. However, they are used as a source of advertising income for Maseru City Council. These signs look like level road crossings, or zebra crossings from the air.
Overhead road signage

Overhead road signage

We intend to use Mapillary much more than this. Future tasks are going to rely on a variety of sources to capture ground truth, and improve accuracy and tag buildings correctly, as well as see roadsigns and walls and fences which sometimes aren’t visible from the air.

Tension, Excitement, Mad Chatter and a Major Milestone

How many mappers does it take to #MapLesotho? Simple answer: All of us!

In point of fact *at least* 250 individuals have made their marks on the Lesothan digital landscape in the Rural Task alone. From a collection of NGOs, a Local Authority, Local Schools in Lesotho, in Ireland, in the US and Canada,, from towns, groups and planners around the world and even his Excellency Paramente Phamoste (who has quite literally thrown his hat into the mapping arena that is Fingal County Council Chamber) we have all made a real map. You have made a real map. We are #MapLesotho.


The Mapping Arena – Fingal County Council Chamber

After a series of successful and fun mapathons, last night, as we pinned our modes, lines and edges, tagged our features and marked our tiles as done, we waited on a lone mapper labouring on tile #2057 – up in top left of the task buried in green and yellow tiles – a lone tile crying out to be mapped and a mapper working his craft. We tweeted with fervour, egging each other on, waiting, waiting, waiting. Sharing our excitement, our impatience, holding our collective breath until finally with a click – @DaCor marked the tile done and we hit a major milestone: 100%!!! – the Lesotho Rural Task has been mapped! The graphs ticked over and the wee thermometer on the Task Manager doesn’t lie. We all pulled together and made it happen. 100% Mapped! It feels great. With a massive push of 7% at last Friday’s marathon we pushed to within 3% of our goal. And then we all pushed on and wish wide contributions we made it happen.

We map with varying levels of detail, some painstaking, some clever, all methodical and all make our own mark. We map in different guises, with funny handles, at different times and in different places – but we all map together. We sit in front of our own screens and interpret the landscape. Sometimes we have the pleasure of coming together in physical surroundings and sharing our passions and enthusiasms. Wherever we are though we never map alone.

As we pat ourselves on the back, all so deserving, much of the real work now lies before us as we move back to the Urban Task (already at 54%) and continue the painstaking task of verifying each others work. But tonight we can rest gratified with a job well done. Tomorrow we can start to share in the benefits of having a solid, workable basemap upon which to build. A map that can change the way people live their lives and see their own world.

Thanks to all and good night.

80% Complete – Nothing like a target of 90% for the New Year

In case you missed it last night a number of #MapLesotho mappers pushed the rural task over 80% complete and 14% validated. We are almost there!

In the spirit of end of year targets, don’t we all love those? We want to break the 90% barrier before the end of 2014! Can you help us? 5 Days. 2% per day. Can we do it!? Spread the word!

Join in!

So please steal some time from your employer / study / family this last week of the year and push#MapLesotho over the 90% mark for the new year.

80% Complete

Mark Your Diary!

There will be a Mapathon taking place on the 16th of January. More details to follow in the coming days.


#MapLesotho hits the magical 2 million openstreetmap nodes

Sounds good, hey?

Creating a whole map of a country is a huge task. And such huge tasks have lots of landmarks along the way.

A node is created every time we click the map, putting the turn of a river, the junction of a road, or the corner of a building. A collection of nodes is a relation, so all four corners of a square building are a relation.


When the Fingal Team touched down in Lesotho in February 2014 the kingdom had just over 25,000 nodes. Most of these nodes were uploads from people who drove around the main highways and  tagged a handful of significant building in Maseru. Some mapping was done near Morija outside Maseru, but little else in the countryside except for the dams, the airports and the centre placemark of the bigger towns. Now with 2,044,582 nodes the country has increased in OSM detail by a factor of 82!

Along the way we have had huge inputs from groups and individuals who are too numerous to mention. The Basotho Planners were the first to go, creating around 100,000 of the nodes in an intensive week in February and concentrating on Ha Foso. Then with the help of Dave Corley in April five #hotosm tasks were set up which easily doubled that.

Then we started using the hashtag #MapLesotho all around the world people joined in and our Polish, Canadian, German and Catalonian colleagues each contributed a big splash of edits at various times in the urban and rural tasks.  The person who I need to mention, because he works regularly and consistently is our learnod G.I.S. teacher Shawn Day who trades under the OSM handle Eireidium. Also worthy of mention is Soren Johannessen who, apart from mapping, watches the OSM system stats and tweets about progress!

Soren has compared the OSM penetration in Lesotho with six similar African countries and this makes very interesting reading because it is now, rightfully, a hotbed of sustained OSM activity. The fact is if node creation is a sound measure of mapping engagement then Lesotho is now the 29th most mapped country on the planet, and the fifth most mapped African country.

The challenge we all have is to balance the short-term appeal of crisis mapping, with that of a more sustained interest in mapping a continent that lacks environmental control, and orderly physical development. All of these facts are acknowledged in the Lesotho Government’s own strategies. And we also may need to “end-run” the mapping process in some countries to show what effect creating a freely useable and re-useable store of spatial data has for African governance in planning and environmental management. That sounds lofty, but something we have the chance to prove if we continue to #MapLesotho.