Analysing OSM data for planning purposes – not so easy

It’s just so easy to add new features and edit those existing on openstreetmap with it’s impressive number of tools including iD and JOSM. Little know though is the vast array of little tools which have been created for extracting and working with that wonderful open geo data.

This last week has been eye opening for me once again, I have been working with a small group of assistant physical planners on how we get hold of an extract of OSM for Lesotho, how do we load this into our chosen GIS tool (postGIS) and finally what sorts of exploratory analysis is possible.

Thanks to the lovely folks at geofabrik we have a daily Lesotho OSM extract. It takes the form of an osm.pbf file.

It was a week of new experiences for all the APPs, they had for one never touched the command line before. Something you really need to know if you want to use osm2pgsql to get your pbf file into your database.

Each day they will download the lesotho-latest.osm.pbf file from geofabrik and open up the command line to run the following:

osm2pgsql -c -d lesotho0902 -U colinbroderick -H localhost --hstore  --slim --number-processes 10 --extra-attributes -C 10000 1602-lesotho-latest.osm.pbf

This creates at least 3 database tables one for each of the geometry types present in OSM:

  • planet_osm_point
  • planet_osm_line
  • planet_osm_polygon

Screen Shot 2016-02-18 at 10.48.29

We are now able to easily perform some high level spatial analysis using the data in each of these tables.

Have you used the command line before? It’s not so easy to get started with right? After a rocky start all our guys are flying along with it now, it’s no longer scary! They are updating the database everyday!

IMG_1259.JPG

We chose to take on the challenge of training the APP’s to use PostGIS for their analysis as it we felt it was the most performant for them to work between periods of online connectivity.

Watch out for some details of the spatial analysis that will be falling out of these five newly trained spatial analysts!

Find out more on how you can get your own Lesotho database on our analysis wiki

Advertisements

Outcome of the #MapLesotho Mapillary Challenge

The challenge was discussed here has been going for 7 weeks. On the 31st August 2015 it ended at midnight. Mappers all went to bed, early for once…. wondering if their score was high enough to rise a place or two. Of course the prize for 10 lucky mappers is a gorgeous smartphone, pre-loaded with several useful mapping apps with help from Eduardo, Peter and Jan from Mapillary.

Peter Neuebauer

Peter Neuebauer – Mapillary

Jan Erik Solem

Jan Erik Solem – Mapillary

What a roller-coaster ride it was for the last few weeks. Positions changed on a daily basis. Mappers engaged in banter in English and Sesotho. Generally the good spirits were wrapped in encouragement, for friends, for district loyalties and for people who were exciting us all by charging up the leaderboard.

Very valuable exchanges took place between mappers trying to tether to the internet using their phones, or using HOTOSM task manager, or even how to draw circular huts (Mokhoros) or using the ford tag. All of this re-inforced previously learned OSM skills.

Screen Shot 2015-08-07 at 11.02.44

Mapillary can be used to add details like posts and road signs to the map of Lesotho

And since we began all previous #MapLesotho records were broken here are some:

  • Basotho based OSM users created 2.9 Million nodes over the period of the competition
  • An average of 54,419 nodes were created by each tracked user. Twelve mappers exceeded this average.
  • Manobby rose from number #48 on week 2, to number #3 on week six, with about 90% of her mapping squeezed into week 6
  • She was only surpassed by Ntebo who squeezed the creation of almost 200,000 nodes all the in the final week
  • Tshedy was the 10th biggest mapper in the world for the month of August. Four of the mappers above her were doing bulk imports, which means Lesotho has an elite mapper
  • On the 28th of August mappers collectively made 252,185 nodes, which is a record for any single 24 hours during over a year of #MapLesotho
  • Lesotho was in the top 5 countries in the world for the number of “modified” nodes on 14 occasions during the 7 weeks
  • The #MapLesotho Urban task had taken a whole year to reach just 68% completeness was finished off before the end of week 4

The end result of the competition is that there are 10 winners, who are seen here. The Final table of the top 10 winners now makes a little Mapillary community in Lesotho who will eventually create a free-of copyright resource used to assist with accurate mapping.

#MapLesotho Mapillary Leaderboard

#MapLesotho Mapillary Leaderboard

Huge lessons were learned from this phase of #MapLesotho. One is that OSM can be gamified. A second is that mobile phones are a reliable, and now affordable source of internet connection. Third is that Basotho Assistant Area Planners are capable of mapping (on average) 1,000 nodes per day. A fourth is that as mapping increases so does the quality of mapping. Fifthly, a critical mass of people mapping creates a community effect, people who are also mapping but not with winning the challenge as as target. Lastly, not all districts in Lesotho are created equal in terms of their enthusiasm to participate in #MapLesotho.

District tallies for the #MapLesotho Mapillary Challenge

District tallies for the #MapLesotho Mapillary Challenge

Finally, on behalf of all judges, contestants and OSM enthusiasts I want to thank Mapillary for supporting this initiative through sponsorship of this phase and the use of their wonderful free resources to help #MapLesotho in the future. And on behalf of our sponsors I want to congratulate all those who took part. Now, instead of monitoring use, animating new users, and I myself can get back to mapping as DeBigC.

#MapLesotho – The benefits to Physical Planners

Lineo Mothae and Ithabeleng ‘Moleli are Assistant Physical Planners assigned to the Districts of Maseru and Mafeteng respectively.

Lesotho 025

Lineo Mothae – APP – Maseru District Council

IMG-20141214-WA0018

Ithabeleng ‘Moleli – APP – Mafeteng District Council


Last February was the first time our partners from Fingal County Council in Ireland introduced OSM as a tool for spatial planners. We can see a direct benefit of the spatial data being added by #MapLesotho each and every day.

IMG_2879

Ithabeleng ‘Moleli talks about the usefulness of OSM:

OSM is used to plot information on the map for streets, sewer lines property boundaries electric, ambulance and fire extinguishers utility lines, parking spaces. Having data of this type makes it possible to carryout diverse tasks in planning; be it extension of road networks etc. OSM helps to map the situation as it is on the ground at the same time updating of maps is easier.

Lineo Mothae details her experience of using OSM for planning:

My first hand experience with using OSM was in February.As a planner in order to make a village layout, a number of factors are considered. One being the direction of growth the development is taking. Using osm makes it easier to decide where plans for for new development could be made as data on osm is accessible and up to date since it can be updated from time to time to incorporate new developments.

Ha’Foso – Our first area mapped in February

OSM to me is an essential planning tool.

Mazenod for example, not much development was depicted of the map but now after Osm there is so much change. I have also seen it is possible to map any area, no local knowledge is required except here and there for description of land uses.

Compared to other options in the past openstreetmap is free and has a more frequent update cycle (always being updated). Planners in other countries are relying on it, and we think we should start to do the same.

Would you like to help us finish mapping Lesotho? We are currently at 76% complete. Join in today and map a tile from our mapping task. All you need is a web browser and some time!

#MapLesotho hits the magical 2 million openstreetmap nodes

Sounds good, hey?

Creating a whole map of a country is a huge task. And such huge tasks have lots of landmarks along the way.

A node is created every time we click the map, putting the turn of a river, the junction of a road, or the corner of a building. A collection of nodes is a relation, so all four corners of a square building are a relation.

nodes

When the Fingal Team touched down in Lesotho in February 2014 the kingdom had just over 25,000 nodes. Most of these nodes were uploads from people who drove around the main highways and  tagged a handful of significant building in Maseru. Some mapping was done near Morija outside Maseru, but little else in the countryside except for the dams, the airports and the centre placemark of the bigger towns. Now with 2,044,582 nodes the country has increased in OSM detail by a factor of 82!

Along the way we have had huge inputs from groups and individuals who are too numerous to mention. The Basotho Planners were the first to go, creating around 100,000 of the nodes in an intensive week in February and concentrating on Ha Foso. Then with the help of Dave Corley in April five #hotosm tasks were set up which easily doubled that.

Then we started using the hashtag #MapLesotho all around the world people joined in and our Polish, Canadian, German and Catalonian colleagues each contributed a big splash of edits at various times in the urban and rural tasks.  The person who I need to mention, because he works regularly and consistently is our learnod G.I.S. teacher Shawn Day who trades under the OSM handle Eireidium. Also worthy of mention is Soren Johannessen who, apart from mapping, watches the OSM system stats and tweets about progress!

Soren has compared the OSM penetration in Lesotho with six similar African countries and this makes very interesting reading because it is now, rightfully, a hotbed of sustained OSM activity. The fact is if node creation is a sound measure of mapping engagement then Lesotho is now the 29th most mapped country on the planet, and the fifth most mapped African country.

The challenge we all have is to balance the short-term appeal of crisis mapping, with that of a more sustained interest in mapping a continent that lacks environmental control, and orderly physical development. All of these facts are acknowledged in the Lesotho Government’s own strategies. And we also may need to “end-run” the mapping process in some countries to show what effect creating a freely useable and re-useable store of spatial data has for African governance in planning and environmental management. That sounds lofty, but something we have the chance to prove if we continue to #MapLesotho.