Making it easier to access and use Earth observation data

Blog / Andrew Newman / June 25, 2020

Share

Earth Observation images of Scotland

Earth Observation images of Scotland

This article has been adapted from Defra digital. The original version of this blog can be found here.

The European Space Agency’s Sentinel 1(radar) and Sentinel 2 (optical) missions make vast quantities of satellite data freely available as open data. At Defra, we recognise these data sources are key to the delivery of better services and ultimately better environmental outcomes. 

But satellite data is big, complex and hard to use, users need to: 

• know where to find the data 
• find ways to access the data via downloads or web services and find storage solutions for the data
• understand how to pre-process the data, for example to correct for distortion caused by the earth’s atmosphere and surface.
• transform the data into formats they could use in their analysis tools and Geographic Information Systems (GIS)

All of this takes considerable amounts of time, specialist skills and significant computer processing power. These issues have often blocked our specialists from using the data and where people did use the data different processing methods led to inconsistency.

Earth Observation image of Lindisfarne

Earth Observation image of Lindisfarne

The Earth Observation Data Service

The Earth Observation Data Service has been built to make all this simpler and faster.The service has 4 core components: 

1. A near real time feed of data from Copernicus via a managed Application Programming Interface (API) service provided by Atos Mundi. 
2. Earth Observation data processing engines that extract, transform and load the data making it ready to use. The processing:
   • produces 10m resolution raster datasets. Sentinel-1 is processed to a backscatter product, Sentinel-2 is a surface reflectance product
   • generates cloud masks and topographic shadow masks that can be applied to Sentinel-2 imagery (images with >95% cloud are not processed)
   • sharpens the 20m image bands in Sentinel-2 to 10m through application of linear regression models.
   • removes the three 60m resolution bands from Sentinel-2 imagery, as these collect information within the atmosphere and are not designed for use in a surface reflectance product 
   • corrects atmospheric and surface distortions
   • ortho-rectifies and re-projects images
   • generates image level metadata
3. A data store containing the ready to use data and metadata, including an 18-month archive.  
4. A website enabling users to filter the extensive data catalogue, create products such as mosaics and time-series, and generate web service links to consume these products in GIS. Users can also download whole datasets or clip them to their area of interest to save storage space and download time. An API provides an efficient way for users to access data programmatically.  

New satellite data is available to users, in ready to use form, within 24 hours of it being captured by the Sentinel satellites.  

The service has been built by SCYSIS (now part of CGI) on Defra’s internally managed Azure cloud environment, it is a complex stack of open source tech including: ARCSI and SNAP for data processing, GeoServer for data storage and sharing and GeoNode for the portal.  

Our ready to use data is open data

Defra group is committed to publishing as much of our data as possible as open data. We have partnered with the Science and Technology Facilities Council and our archive of data can be found in the CEDA archive. The data can be reused, by anyone, for free, under the terms of the Open Government Licence.

The Earth Observation Data Service has been developed for use by Defra and its agencies. It puts Defra at the forefront of government innovation in earth observation data. Other public sector organisations have shown an interest in the service and we are working with the GeoSpatial Commission to explore how this might be achieved.  If you are a public sector organisation and are interested in joining this work, please email us at open@defra.gov.uk.  

Expected benefits – what Defra will get from the service

The Earth Observation Data Service will help staff across Defra Group with their work, delivering environmental observations, powering models and enabling new analyses. The service will enable us to deliver existing services more effectively and will provide new insights. We know the data will be used to:  

• support emergency responses (for example, flooding)  
• measure changes in the extents of habitats  
• monitor bare soil and other indicators to understand environmental quality
• map crops to provide evidence to support agricultural policies
• develop new approaches to monitoring landscapes that would have been prohibitively complex and expensive in the past, for example monitoring consistently across large upland areas

As we adapt our ways of working because of coronavirus we expect the demands for earth observation data to continue to grow. For example, we are looking at how we can target field visits more effectively by making more use of earth observation data.

This new service will enable us to do all these things more quickly or frequently and with less specialist input.  

Thanks to Paula Lightfoot at the Joint Nature Conservation Committee (JNCC) for her assistance with this post. You can read more about earth observation in the Spring issue of ‘Nature News’, the JNCC newsletter.

Andrew Newman is the Workstream Lead, GDPR Programme at Defra.