Spatial Data: the jam in the GIS cake
Duration:     0:27:29

START AUDIO

Voiceover: The Spatial Jam, an Esri UK podcast.

Beth: Making your life a bit easier in terms of working with data

Liz: Sourcing data that's reliable.

Elleni: And then the difference between me giving up, throwing my laptop out the window.

Hannah: the style, such as like the pencil style or the newspaper style. And I've found that they're really good for community engagement. Just adding those in from the Living Atlas can really make such a difference with not a lot of effort.

Chris: Anything where you would traditionally search by distance, but actually searching by time gives you much more relevant results.

Beth: Welcome to the Spatial Jam. I'm Beth and today I'm joined by Elleni.

Elleni: Hi everyone. We've got a very special episode today, which is all about content.

Beth: And it's not just myself and Elleni today. We have two special guests with us. First up, we have Liz.

Liz: Hi everyone, I'm Liz De Guzman and I'm the Data Curator and I'm really excited to be here today.

Beth: And we also have Hannah.

Hannah: Hi, my name’s Hannah Fieldsend and I work in the content team at Esri UK. As an added bonus at the end of this episode, I got chance to interview Chris from TravelTime, one of our data partners.

Beth: Great. So, as Elleni said, today's Spatial Jam is all about spatial data. So, something often referred to as the fuel for the ArcGIS system. But as regular listeners will know, we're less about cars and more about cake on this podcast. So, I'd prefer to liken it to the jam in the GIS cake. Now, Elleni, any other metaphors you like to use?

Elleni: Yes, I would say data is more the base of the cake, right? It's the first thing that you use in GIS. It's the first thing that you have to wrestle with in any kind of GIS workflow to get to the jam, which is then the map output, in my opinion. (Laughter) But we could probably sit here all day talking about different metaphors.

Beth: Yes, that's really true. We could probably spend all day talking about cake as well. (Laughter)

So, let's move on. So, data is different things to different people. Elleni, what does it actually mean to you?

Elleni: I guess, data is something that in my role as a GIS analyst and GIS consultant. It is the foundation of everything that I do. But really what data means for me is the difference between trying to actually be able to answer a question and answer it accurately.

And then the difference between me giving up, throwing my laptop out the window and not actually (Laughter) trying with it anymore, simply because I think anyone who is a GIS user will know that a lot of their time is often spent wrangling with data, trying to get it into a format that is right for GIS, but also make sure that it's accurate.

For me, it is that base of a cake, of the GIS cake, it’s absolutely fundamental and being able to have the skills not as just as a GIS person, but also as a data engineer or as a data analyst, is actually critical in that role in GIS.

Beth: Yes, that's so true. And Hannah, any thoughts on what Elleni is saying?

Hannah: Yes, definitely. So, personally, data to me means, you know, my personal data from apps on my phone and things and stuff that third-party companies get about me. And when we're talking about spatial data, those two things aren't necessarily disconnected.

There's loads of different types. We're not just talking about environmental and utilities and things. There's demographic data, physical location of devices on the ground, spending habits and all of this can be used in GIS.

Beth: So, Elleni, you talk about data wrangling and how it can be annoying and make you want to throw your laptop out the window, which I don't recommend, by the way. But what are some of the things that you've learned over the years about making your life a bit easier in terms of working with data?

Elleni: It's been trial by fire, I think, sometimes being able to first think about the challenge that I'm trying to approach and then, obviously, finding the right datasets for it. And once that first step has been overcome, which is often the most challenging part, even just finding the data, it's then about using the tools that I have available to me to try and simplify that dataset or put it in a format that ArcGIS can read.

And a lot of the tools that I use, actually, are simple tools that a lot of us have on our laptops anyway. So, Excel is a great tool. I might do a little bit of work, just to make the field names a bit nicer, making sure, you know, they've all got capital letters, no spaces, no weird special characters. And once I've done all of that, if I'm positive that that dataset has all the information I need to then answer that challenge or that problem that I'm trying to solve, then I'll bring it in.

Sometimes, however, it does mean I have to calculate additional fields. So, I might want to add in a couple of rows or columns to capture that, actually in GIS itself. And doing that in Excel can actually speed that process up really, really quickly. So, when you bring it into a tool like ArcGIS Pro, for example, it's there and ready to go.

It's formatted in the right way, the date fields as read as date fields and it's all fine. You know, this is a huge topic. It's probably an episode in itself. But another thing to think about is once you've wrangled that data and you've got it into a useable format, using tools in ArcGIS Pro like domains and attribute calculations, all of these things that maintain the data integrity, maintain the data quality when you move forwards with that data.

You want to make sure that it has all those rules in place, so that it's maintained going forwards. Then you don't have to wrangle it (Laughter) every week or every month when a new update is made to it. So, just some things to think about there.

Beth: I think that's a really good point, because I think you can really overlook those initial steps and think, "Oh, I'll just get this in and get started with it” and then, you know, a few months down the line, you think, “I really wish I'd just spent that little bit of extra time at the start, because now I have to fix all of these problems." So, yes, definitely worth putting in that work at the start.

Liz: Just to hop in there, Elleni, just with your point about sourcing data that's reliable. I think with the growth of open data and a lot more people are sharing data now, now that there's more open data, there's more resources. So, one of the resources that I use in my role is that the Geo Spatial Commission published a guidance and they've listed, basically, different geo spatial themes and they've listed the data publishers that provide that data.

And they have an extra column, and it shows whether that data or that data supplier is authoritative is not. It's just to make a point that there are resources out there to help people who are sourcing data and data wrangling.

Elleni: That's really cool. I'm definitely going to check that out after. Thank you Liz.

Beth: So are there any new tools that you've been using in Pro or anywhere else in the system that have been really helping you with this work recently Elleni?

Elleni: Yes. So, actually, there's a few, and I guess, it completely depends on the type of dataset that you're using. I think the important thing to know in ArcGIS Pro is whatever dataset you're working with ArcGIS Pro is pretty clever in knowing what it is.

So, if you're working with LiDAR for example, it would bring up LiDAR classification tools. If you're working with CAD data, it will bring up tools for geo referencing, that CAD information getting it into the right place.

The same with CSV information, if you drop a CSV file into there, when you right click that, it has a tool to enable X, Y- to add X and Y points to make it point layer. So, use ArcGIS in terms of its capability of intuitively knowing what dataset you're using. But, also, I would suggest checking out the data engineering tools as well on ArcGIS Pro.

I think it came out around 2.8. It's a very useful tool to, kind of, actually, not just look at field names and whether they've got weird characters in there, but also look at, actually, the data quality itself captured in those fields, in those attributes.

So, yes, those are the sorts of tools that I'd say use ArcGIS Pro in its native, wonderful ability to know what you're working with, but also check out data engineering tools.

Hannah: I don't know whether I should be admitting this, but I've started again with some people at work, where if you see a car reg with a data format on the end, you need to send it to the group… (Laughter)

So, someone sent me a dot DAT

Elleni: That's brilliant.

Hannah: I've got a DWG in there as well. So, tonnes of things that can be read by Pro.

Elleni: There's too many. The list, honestly, is huge.

Hannah: I'm after a shape. I want SHP. I feel like that's the Holy Grail.

(Laughter)

Beth: I don't know where to go from there.

Some of the new tools that I've heard about recently for Pro is that you can actually now bring in PDFs and convert them to TIF Fs and put them in place. So, you're geo-referencing them.

So, you've been able to do it for a while with a normal TIFF and bring it in, so maybe you've got like a site plan or, you know, old historic imagery that you want to put over the top. But, yes, now you can do it with PDFs as well and you don't have to transform everything before you can bring it in.

It's all part of the tools that are just available out of the box, which is really great.

Elleni: That's actually a really good point, Beth, about- or recognising that all of us as GIS users won't necessarily know what datasets are supported by a tool like ArcGIS and for that I would say, if you're ever unsure check out Esri Community, check out the documentation for ArcGIS Pro for supporting data formats, because I think you'll be pleasantly surprised with the amount of data that it does support.

And if it is something that you need to transform then, often, there's a community of users that have already figured out the answer for you. So, definitely check that out.

Beth: So, we've been talking a lot about data sourcing and wrangling and everything, but a lot of people don't have the time, desire or the skills to actually do that themselves. And there are a lot of resources out there to help you. Elleni, you've talked about some of the help forums and things like that.

But something that I have seen people do before is, kind of, go through ArcGIS online and see what is available there. And, I guess, this is just a bit of a warning for those that are out there. Just be careful if you are looking on ArcGIS online, because anyone that's got an ArcGIS online account can just post anything they want on there. So, you can find some interesting things on there. Some different content, you know.

Some of it might be exactly what you're looking for and might be from an authoritative source. It might have all of the right information in there, it might be up to date. But at the same time, it could be something that someone has downloaded from somewhere, manipulated, they might have put some false information in there, because they're just using it for testing.

I guess, it's a warning just to be sure of where you're getting your data from. I guess, that's a good point, how can you actually find data that's already been processed, that someone's done all of this work for you. And Liz, you're the data curator here at Esri UK, you work with the Living Atlas. Do you want to tell us a little bit about what it is and why does it even exist?

Liz: Of course, so my role, I guess, is to basically live and breathe Living Atlas and make sure that our users get data that is of the highest quality, that's ready to use and it's authoritative. So, Living Atlas, for those who aren't familiar with it is basically a library of curated, authoritative and ready-to-use datasets.

So, me, as Esri UK's data curator, I curate UK datasets that's been contributed by organisations and the great thing about Living Atlas is that it's a subset of content in ArcGIS and you can integrate it across ArcGIS online, ArcGIS Pro and even some of our applications like Field Maps and StoryMaps, for example.

And it's a really great resource that not many people know about. You know, globally we have over 10,000 items that's been shared by Esri, our partners and the global GIS community. And, you know, each of these items has been reviewed by a data curator, like myself to ensure that it's the best that it can be for you, our users really.

Beth: I love the Living Atlas. There's so much great data in there, and not just data, but apps and things like that as well. But what's new? And what's coming up in the future?

Liz: What's new? So, I've been trying to source more datasets. Our new data contributors include the Scottish Government and also NatureScot. So, the Scottish Government have recently contributed their Scottish index of multiple deprivation for the year 2020 and we've also welcomed NatureScot who have contributed their protected areas dataset.

So, things like sites of special scientific interest, national nature reserves. There's so many things that you can do with these new additions. And one of the things I really like as well was last year we've made pre-trained, geospatial deep learning models available.

So, you know, people can now just implement this in their GIS workflows, these models that's already been pre-trained to help you identify things like building footprints or perhaps land cover.

Beth: I think that's really great, because I think, you know, across both deep learning, someone could say, "Well, you know, I can set that up myself, that's fine. I can do it." But not everyone can. And it's also things like, you know, bringing in the sites of special scientific interest.

Lots of people out there that are listening to this will probably be processing that. They'll be using it, but think of all the time we can save, if everyone's just using that same initial source.

Hannah: I was just going to say, also, really great to hear Liz placing focus on more regional datasets. It can only be a good thing to get more authoritative data from Scotland and Wales.

Beth: So, Liz, why would anyone actually contribute their data to the Living Atlas? What's in it for them?

Liz: For the customers and organisations I've been working with, it's just another avenue to grow access to their data. Not many people know that, you know, if you have data already sitting in ArcGIS Online or in your portal, that you can actually really easily contribute it to Living Atlas.

So, if you're interested in contributing your data, visit the Living Atlas website, I'd say. There's lots of resources there that show you how to nominate your data.

Beth: I think all of us on this podcast have used the Living Atlas a lot over the years. So, what's everyone's favourite dataset? You can only have one?

Hannah: Hey, I wasn't prepared for this Beth.

(Laughter)

Beth: Off the top of your head, what is your favourite. So, I'm going to go first, so that no one steals it. I think mine is the new Land Cover 2020 map…

Elleni: Classic.

Beth: …because it's, you know, an amazing dataset. It's high resolution land cover for the whole world and it's available just with the click of a button. Elleni, how about you?

Elleni: Yes, my favourite dataset has to be the World Traffic Layer.

Beth: Oh, good one.

Elleni: The live layer that gets updated every, I think it's every five or…

Beth: Five minutes.

Elleni: …fifteen minutes. It's five minutes, yes. But it's really, really cool, because not only do you have that map service, where you can see in red, amber or green the state of what traffic is like at that current time on a particular segment of road.

But you can also see where there's live construction works going on, which in my, sort of, field that I work in, it's really important for utilities to know where construction might be going on that might prevent them from doing their own construction on utilities that are underground.

So, that definitely would be my favourite one. It's a really cool dataset.

Beth: I hadn't even thought about that use case with that, so that's really good to hear about, actually. Hannah, what's your favourite?

Hannah: Probably an easy cop-out, but the basemaps. Some of the basemaps that the Esri Inc team work on, I think it's- not a lot of people use the style, such as the pencil style or the newspaper style. And I've found that they're really good for community engagement and stakeholder communication and things, if you're making a story map, just adding those in from the Living Atlas can really make such a difference with not a lot of effort.

Beth: Also, one really important one that they've brought out recently, it's actually a couple is the accessible basemaps.

Hannah: Definitely, yes.

Beth: So, for people that have visual impairment or dyslexia and things like that, it's just basemaps that make it easier for them to them, actually, understand the map and what's going on in it. So, you know, for those of us who don't have those visual impairments, maybe we don't necessarily really think about it day-to-day.

So, this basemap that you can just bring in the style and just use it and just make life easier for others, I think that's a really great inclusion that goes with yours Hannah. And Liz, finally on to you?

Liz: I'm going to be a bit rogue. So, as you mentioned, Beth, earlier, the Living Atlas isn't just data layers, it's also applications. So, one of my favourite things is the ArcGIS Living Atlas Indicators of the Planet application. So, this is an app which displays real-time information on things like wildfires or droughts, let's say. I think it's just an amazing resource, where you can view, sort of, these indicators over time through things like graphs, maps and you can even access resources about new research.

Beth:  The Living Atlas contains loads of great open data, there are some things in there that, you know, require credits to use. But we also do have premium data that goes a little bit beyond that, you know, where things aren't necessarily available for everyone to use, they've got licensing models setting behind them.

We have solutions for that, things like maps. Plus, for mapping we have National Data Service for Statistics, but we also have a lot of third-party data suppliers as well, that are providing us with maps and apps to help our customers to do their everyday work and to go that little bit further.

So, Hannah, working in the content team, you work with our premium content partners all the time, do you have any examples of great things that our partners are doing at the moment?

Hannah: Yes. So, in my role, we work with a number of different premium data providers, third parties, so TomTom, HERE and the OS for example. And along with this, we work with some smaller content providers or ones that we're just starting, developing relationships with.

For example, Cyclomedia have some really cool, on-the-ground 360 imagery and TravelTime as well. They're a really cool company that we partnered with last year. So, lucky for me, earlier in this week, I got the chance to catch up with Chris Hutchinson from TravelTime and he very kindly agreed to let me interview him for the podcast.
Welcome Chris.

Chris: Thanks very much, Hannah. It's really nice to be here.

Hannah: Thank you for making the time to come along. So, let me start by asking you what is TravelTime and what is TravelTime's content?

Chris: So, TravelTime is an API and a set of integrations into other tools that enables the world and spatial data to be searched by travel time instead of by distance and crucially we make this available for any mode of transport. So, not just looking at driving times, but walking, cycling and also public transport.

So, what this looks like is - instead of calculating an A to B, what we enable users to do is calculate A to every possible B. So, asking questions like where can I get to within 45 minutes by public transport, leaving my home at 9:00am? And for our work with Esri we've enabled this functionality to be accessed directly within ArcGIS Pro.

Hannah: So, it's more following a network rather than as the crow flies, distances?

Chris: Yes, exactly. People can't move as the crow flies. We can't move in a straight line. So, looking at things in terms of distance, is not always that helpful and time is a much more valuable metric.

Hannah: Can you tell me a bit more about how the data's brought together and how it's created?

Chris: Yes. So, looking at the public transport side, in particular, what we do is we go and hoover up all of the public transport agencies timetables and data and we, kind of, consolidate that all into one holistic view of the public transport network. And sometimes this can be quite straight forward, but most of the time this involves going and processing this data, turning it from, sort of, formats like a PDF of a timetable into a nice clean GTFS data. So, we can then use that in the model.

And in some countries it's nice and easy. Some countries this is a massive part of what we do. One of the biggest focuses for us and challenges that we've always had to overcome is making the technology as performant as possible at scale. We're now processing over 50 billion locations a month through the API and the performance and the response times is crucial for what our clients are using it to do.

Hannah: Great, okay. So, what kinds of problems are users combining it with ArcGIS to solve? Do you have any of your favourite applications of the content?

Chris: Yes, absolutely. So, the range of different use cases that we see clients use the technology for is super broad. Almost, every week, (Laughter) I come across a customer using it in a way that I've never even thought of before. Because in a way changing from distance to time is quite a subtle change, but it can have a massive impact.

So, for example, the impact on a, sort of, individual level might be a patient being allocated to their nearest hospital or their nearest COVID testing centre, maybe, based on how long it will take them to get there, instead of distance. And this might mean, rather than being sent from Portsmouth to the Isle of Wight, you'll be sent somewhere where you can actually get to.

In terms of, kind of, impact at the business level, a lot of the decisions that our clients are using the data for are decisions that are going to have a really long pay-off period. Things like where to invest in a new warehouse or where to locate dark kitchens across the city.

These are things that using our data, clients can make, kind of more informed decisions and, yes, the benefits of those can last for a really, really long time.

Hannah: Do you mind telling me what a dark kitchen is, Chris?

Chris: Yes, absolutely. So, yes, a dark kitchen is, basically, if you were going to order food from delivery or Uber Eats, it may not actually come from an actual restaurant. A lot of these restaurants operate what's called a dark kitchen, it's a facility specifically for producing food for deliveries.

Hannah: I feel like my whole life has been a lie. I never knew that.

(Laughter)

Just as an add-on to your last point. I work with TravelTime data quite often and, personally, my favourite use case that I've seen was working with a retail company who had sustainability at the forefront of their development within the organisation and they were using it to locate their existing staff locations in a certain area that worked on a shift basis.

And then trying to communicate the different routes that they were currently using to get to the warehouse and using to leverage local bus service providers to put on more routes in some of those hard-to-reach areas.

Chris: Sustainability is actually really common in a lot of what we see. Because enabling businesses to use public transport data and multi-modal data, like we have a driving and train model, for example, enables them to make decisions not just based around driving times.

Hannah: So, in terms of future development where can you see TravelTime as an organisation or the content, where can you see development going?

Chris: So, one of our key philosophies, I guess, is to make sure our data and our tools are available wherever someone might want to use them. And in terms of the, kind of, Esri stack, we currently have an integration into ArcGIS Pro, but we want to make sure that it's useable for all Esri users, whatever tools they might be using.

So, for example, we're looking at ArcGIS online integration, which would enable maybe those more, sort of, business users to use our tools. The pro add-in is great for doing those more complicated pieces of analysis that require like really deep GIS knowledge, but our technology is equally valuable at answering some of those maybe more high-level business questions around, you know, salesforce planning or marketing and that kind of thing and having an integration into ArcGIS Online would enable those business users to get value out of a tool.

Hannah: Definitely and we've such a huge uplift in software-as-a-service offering. So, the fact that you guys aren't just a content partner in that we deliver travel time data to our users, but you're also working with us to integrate into the platform. It's a really valuable partnership.

Chris: Yes, absolutely. And that's the benefit of it being an API is that the API can be used, you know, in any different tool, anything where you would traditionally search by distance, but actually searching by time gives you much more relevant results.

Hannah: Great, thank you very much.

Elleni: That was a really interesting discussion from yourself, Hannah and Chris and massive thank you to Chris for going into some of the details of the applications of that data. And I think, really, what we've learned across this episode is there's no limits to your imagination in terms of how you use data to solve a particular problem.

Absolutely it comes back to my point, right at the beginning, about data is definitely that base to the GIS cake, the jam element is, perhaps, doing the data wrangling and getting it to that perfectionist piece, so it's that perfect data, which is the jam bit. And I'd say the icing is about then how you apply that and share that to all of our users in things like the Living Atlas where people can make more use of it.

Beth: Thanks to everyone for listening. We'd love to hear what you think, so please do get in touch at podcast@esriuk.com and don't forget to subscribe and rate us on your chosen podcast channel. It really does make a difference. We hope you join us again.

Voiceover:  The view of the presenters may differ from those of Esri UK.

END AUDIO