I recently read a blog post about using Yahoo pipes on ArcGIS online to map out Flickr photos. That was the first time I’d heard about Yahoo pipes. This got me curious about what pipes actually were. So I did a little bit of digging around, and found that they were actually a very useful set of tools. Pipes essentially enable you to ‘mash up’ and create new content dynamically over the web, akin to ArcGIS online, except with any content (including non-geographic). This is not a new concept – speak to any web developer, and they’ll tell you how they’ve been doing this for years. However, the crucial advantage that Yahoo Pipes gives us is the relatively user friendly interface that allows the user to build up these ‘pipes’. I think it’s quite comparable to the model builder interface on ArcGIS desktop, where a user can drag and drop tools to put together a single geoprocessing task.
The end products of these pipes are web services, containing simple csv files or KML/RSS feeds. These urls are dynamic, which means the contents are constantly updated. A lot of pipes are not related to any mapping content – they are merely a mash up of various RSS feeds, or an updated listing of TV programs from various websites. However, pipes can also contribute to some really good, dynamic content for ArcGIS online. In this blog post, I’ll take you through how I created a useful pipe for News feeds from the BBC website.
There are a lot of good news feeds served up by the BBC as RSS feeds, and one such feed is the World news one. This feed has all the top headlines from around the world in an RSS feed. One of the drawbacks of having this data as an RSS feed is that we can’t put it on a map. However, we can use Yahoo pipes to do that clever bit of processing to convert it into a GeoRSS feed so that we can use it on ArcGIS online. The first step is to import our feed into the Pipe:
We use the text input to get the URL as text; We then supply the url to the news feed in the ‘default’ field. This gives us a link to the BBC news feed so that it can be passed into the pipe.
We then need to pass this url through a geocoder that uses the place name in the RSS feed to give it a lat and long. In this example, I use the geonames service, which is a free Global city level geocoder. In order to use that service, I just need to add a bit of text at the beginning of the url – I can use the url builder module to create my url, which then sends the information to geonames and receives a lat and long against each record.
We can then use the fetch feed module to bring that feed back into Yahoo Pipes, and finally use the output to generate a dynamic link to my brand new GeoRSS feed!
On the Yahoo pipes run page, I can see a yahoo map with my geocoded news articles:
To use this in ArcGIS online, all we need to do is grab the link from the ‘more options’ menu, and Right click on Get as KML, and copy the link location. If we then switch over to ArcGIS online, we can use the ‘Add layer from Web’ option to add this dynamic KML layer. If you need control of the symbology, then we can add it as a csv file. All we need to do is change the url from ‘=kml’ to ‘=csv’:
Once we hit add layer, voila! We have a dynamic GeoRSS feed on ArcGIS online (with some nice symbology to boot!):
Once it is on ArcGIS online, we can add other content (either from ArcGIS online or from your own geographic data) to add more intelligence to this webmap. This webmap will only work as long as it can establish a connection to the Yahoo Pipes service.