Open Data Cube in SatSure: By Thaiseer Parammal

Remote Sensing is nothing but observing and analysing objects from a distance, using a sensor usually installed on a satellite called earth observation satellites, and recording this information captured on a timely basis to study the earth. We call it Earth Observation (EO).

Some of the freely available earth observation data are ESA’s Sentinel sensors based satellite images, NASA’s Landsat or Modis based data products. These freely available data products have made access to earth observation data for industry experts, academic researchers, and institutions much simpler. This is making way for breakthrough developments in how earth observation data is perceived and it’s usage and applications. These data and the applications or insights they bear have tremendous potential to deliver breakthrough decision intelligence which can have a high impact on agriculture, infrastructure, climate change, or any other verticals.

With each passing year, the increase in EO satellite data volume is rapidly increasing along with their global coverage. The applications and insights built on top of these are mainly driven by technological advances and open data policies adopted by governments and space agencies. Availability of these data is no longer the major bottleneck for this data’s potential to deliver these high impact insights. The pipeline of data connections from the satellites to the insights engine to the end-user resulted in most chunks of the earth observation data being underutilised.

Under the Committee on Earth Observation Satellites (CEOS) leadership, the Open Data Cube (ODC) initiative was started to address the above problem. Founded in 1984, this international organisation acts as the primary forum for global coordination of space-based Earth observation. The Open Data Cube initiative aims to provide a solution in the form of a data architecture that adds value to its users and increases EO satellite data’s impact. 

The Open Data Cube (ODC) is an Open Source Geospatial Data Management and Analysis Platform that helps you harness the power of Satellite data. According to The Open Data Cube Initiative, its main objective is to increase the impact of satellite data by providing an open and freely accessible exploitation tool and to foster a community to develop, sustain and grow the breadth and depth of applications¹. It presents a common analytical framework composed of a series of data structures and tools that facilitate the organisation and analyse large gridded data collections. 

In SatSure, we harness Open Data Cube’s power to scale the insights-delivery pipeline of our geospatial analysis derived from the historical EO data on a pixel level granularity.

Why Open Data Cube?

Spatial Data Infrastructures (SDI) were created to properly discover, access, manage, distribute, reuse, and preserve storage for these big data sets. Some of the most commonly used and popular platforms among them are:

• Open Data Cube (ODC), 
• Google Earth Engine (GEE), 
• System for Earth Observation Data Access, Processing and Analysis for Land Monitoring (SEPAL) and 
• OpenEO

In the article, An Overview of Platforms for Big Earth Observation Data Management and Analysis, these platforms were evaluated on the key metrics of performance, scalability, replicability etc., with ODC outperforming all of them.

On comparing the data access interoperability and extensibility (Fig. 1) of the platforms, ODC outruns the others. Fig. 2 is an illustration of processing scalability vs the storage scalability of the platforms. All platforms perform high on storage scalability, whereas ODC and GEE are on medium and high in terms of processing scalability. Upon comparing the Open Governance and extensibility in Fig. 3, of all these platforms, ODC is ahead of others by far.

ODC uses standardised schemas to define the products/datasets available, using this information to index the datasets available for a product and keep the metadata in a structured format in the index database. The index database is powered by PostgreSQL. These metadata will have information about the actual location of the Cloud Optimized GeoTIFF (COG) corresponding to each dataset.

Based on the user query, ODC makes a lookup in the index database to determine which datasets correspond to the query and groups them based on the request. Upon receiving this information, it requests the cloud storage to fetch the actual COGs and generates the required images or statistics.

The requests can be in terms of map services like WMS, WMTS, WCS or statistics based WPS requests. Using ODC in the backend, users will be able to tune the queries for specific requirements and fetch the underlying data from a vast collection of datasets available underhood with ease.

The queries can be modified to fetch the products like below on a real-time basis:

• Point or boundary-based extraction
• Applying resampling techniques
• Applying different projections
• Fetching based on time information 

For example, a query optimised to fetch the underlying image of a specific product/region/ time on a visualisation layer will look like the following.

In SatSure, we have satellite data from different sources like ESA’s Sentinel sensors and NASA’s Modis products. We pull these data from their source archives into our AI/ML analytics engine. Before this is fed into our engine, the data undergoes a preprocessing series to be analysis-ready.

In the insights engine, there are proprietary machine learning models and remote sensing semi-automatic algorithms. They generate insights and transform these earth observation data into directly actionable products for solving high impact environmental, economic and social challenges. These products are delivered to end-users in different sectors using various technologies, including Insight Dashboards, RestAPIs, WMS layers, etc., in our delivery engine. This engine is powered using high availability and highly scalable modern architectures of Kubernetes services in our cloud, as described above in Fig. 5.

The Future Ahead

In the wake of five years into the UN’s Sustainable Development Goals (SDGs), the transformational vision and new data requirements called for to realise the 2030 Agenda has only been partially realised. The extent of this challenge has been underestimated and is further amplified by knowledge and availability of geospatial data. The SDGs are highly dependent on geospatial information and Earth observations as the primary data for relating people to their location and place.

In the 11th meeting of the Inter-agency and Expert Group on SDG Indicators Working Group on Geospatial Information (WGGI), a vision was created to see geospatial and location-based information to be recognised and accepted as official data for the SDGs and their global indicators.

At SatSure, by harnessing Open Data Cube’s power, we are ready to scale the United Nations’ vision in achieving SDGs by enabling people’s access to geospatial information at various levels and by democratising modern location awareness requirements using EO datasets.

Written by Thaiseer Parammal, VP of Engineering, SatSure. This article was originally published in The SatSure Newsletter(TSNL)