Jan 8, 2020
10 years of the London Datastore & thinking on city data for the next decade
London Datastore Manager Joseph Colombeau on how the London Datastore has gradually evolved as driver of city-wide data collaboration and innovation, and what our next steps are...
Ten years ago today, City Hall launched the London Datastore.
At the time the Datastore was one of the first of its kind developed by a major city anywhere in the world, making freely available huge amounts of data about the capital. Since then this data has been used to tackle some of London’s most important challenges, such as easing road congestion and improving air quality, as well as providing a wealth of accessible information about how the city functions.
Today the platform currently has around 60,000 users each month and is home to more than 6,000 datasets — up from around 500 when it was first launched in 2010. Initially the Datastore was used primarily as a tool to improve transparency and accountability. Information on what the Greater London Authority (GLA) spends and data that underpins mayoral strategies were published to allow the public to scrutinise the decisions and activity of the Mayor and the GLA.
However, very quickly it became apparent that opening up the city’s data could enable a whole range of positive outcomes beyond simply greater civic transparency.
Today policymakers at City Hall are increasingly seeing data as a means to address some of the city’s challenges through maps and apps open to the public.
- London Rents Map, which 85,000 Londoners used last year to help them find an affordable home
- Schools Atlas, which allows for school place planning as well as being a tool for parents selecting schools for their children;
- Cultural Infrastructure Map, which helps people enjoy and preserve music venues, studios and community halls in their neighbourhoods;
- a range of air quality mapping using data from a network of sensors showing Londoners pollution levels in their local areas, as well as prioritising new electric bus routes.
- Minimising disruption from roadworks: The organisations that manage the city’s energy, transport and water infrastructure use our Infrastructure Mapping Application to share data with each other, allowing them to better target investment and minimise disruption by co-ordinating streetworks.
London’s population is estimated to grow to nearly 11 million by 2050 and data, such as that available via the Datastore, will become even more important as the City Hall works with its partners to positively impact Londoners’ lives.
At their heart, all of these initiatives are about making data accessible to the people who can use it to make a positive difference to our city.
A change in scope?
While the Datastore has allowed us to achieve a lot with open data, actually this is just the tip of the iceberg when it comes to all of the data that is collected, used and shared in London. With this in mind at the back end of 2019 we’ve been exploring how the Datastore can support the sharing of data that isn’t necessarily suitable to be published for public consumption.
Additionally the type of data that the Datastore is being used to share is changing. In 2018 the Datastore gained the capability to share ‘secure’ data (restricted and/or licensed data including permission, privacy, publication and distribution; as well as data that is presently held privately).
With the adoption of sensors we’re starting to see live feeds of data being shared through the Datastore, a change from the usual static tables of data in a CSV or excel format. An example of this has been the EU-funded Sharing Cities project where the GLA and Greenwich council are working together to trial new technologies such as smart lampposts and energy management systems in council housing. These data feeds are then shared privately between the different organisations working on the project.
With new use cases such as these, the Datastore can no longer be considered simply a repository of open data. So in 2019 we undertook a discovery exercise with help from the Open Data Institute (ODI) to help us redefine the Datastore’s evolving mission.
Thinking ahead
This discovery (read full report here) was an opportunity for us to ensure the Datastore aligns with our wider ambitions for city data.
The Smarter London Together roadmap identifies strengthening the city’s ability to work collaboratively as a mission for London’s public services. By taking a more joined-up approach we believe London’s institutions can tackle some of the city’s big challenges using data.
Our hypothesis going in to the discovery exercise was for the London Datastore 3.0 to act as a “central register” for London datasets (linking to data holders through meta-data/descriptions) enabling better sharing across London’s 32 boroughs, agencies, business and research institutions.
Alongside this sit vehicles for agreeing data standards, creating data sharing agreements and helping to set problems-to-solve (e.g. zero carbon city, imporving air quality and other city-wide priorities).
Effectively this would mean changing Datastore from a product to a full-blown data service to better meet the needs of users looking to work collaboratively on data projects.
The Discovery
Over the last three months of 2019 we listened to hundreds of users in London’s data community — including both users and non-users of the Datastore coming from local authorities, academia, the private sector, civil society, think tanks, media organisations, the NHS and other public sector bodies — to explore their needs when it came to accessing data to do their jobs.
Users indicted that the platform does a good job of making London’s data more accessible — although there is room for improvement, particularly around search and navigation as well some of the functionality around sharing secure data. However, many of the difficulties that people face cover a range of cultural and process barriers.
Many of London’s institutions have recognised for several years the value in opening up and collaborating with data. However this common understanding has not always translated into action.
Our research identified the two largest barriers to this was a lack of clarity on what standards should be adhered to (i.e. what “good” looks like) and a lack of leadership to pull people around a common challenge.
So beyond making some technical improvements to the Datastore platform to ensure it is fit for purpose as a central registry of London’s data, the discovery findings have opened up a wider conversation around what guidance and leadership is needed to ensure we can deliver positive impact with the city’s data.
Next steps
Today the ODI have published their discovery report summarising their research and sets out their recommendations for ensuring the Datastore meets the needs of our users and supports our wider ambitions for city data as set out above.
We have synthesised these recommendations into four actions (platform development, content management, standard setting, and community building) and mapped these onto three projects through which we can deliver them:
· London Datastore 3.0 Development
This project will aim to address some of the platform specific problems that the ODI identify. Some of these are specifically around technical features such as search while others are more about quality of content such as metadata.
· GLA Data Strategy
Before looking outwards we need to get our own house in order. Our strategy will look at how City Hall should set up to deliver services with data in a way that maximises public benefit and supports the organisation in realising its goals. We will set out the principles and good practice that the GLA should follow and identify the mechanisms required to do this (such as the governance, skills and technology needed).
· Towards a London Data Strategy
As touched on before, a London Data Strategy should set out a pathway towards a London whose institutions are working together to tackle some of the city’s biggest challenges using data. This isn’t just about shared data but shared goals, shared standards and shared values.
We won’t be starting from scratch as much good work has already started with initiatives such as the London Office of Technology and Innovation (LOTI).
We’ll also be taking great interest in the work of London First’s Data Commission over the next six months and will welcome their recommendations later in the year as an important contribution to this strategy.
We will be communicating more about these projects as they progress in the coming months. If you would like to stay updated by email sign up here.
And finally, thank you!
We’d like to thank everyone who has contributed to this discovery project over the last few months. All those who took part in the workshops, interviews and took our survey provided important insights without which we would not have a clear understanding of the issues that need addressing. We hope to work closely with many of you as we take on this exciting new programme of work.
Finally, a special thanks is warranted to the team at the ODI for delivering this project. In a short space of time they have managed to synthesise a complicated data ecosystem into actionable recommendations that should help us shape the way we do things for the better.
