Australian Geoscience Data Cube

The Australian Geoscience Data Cube (AGDC) is an approach to storing, processing and analyzing large collections of Earth observation data. The technology is designed to meet challenges of national interest by being agile and flexible with vast amounts of layered grid data.

The AGDC reduces processing time of traditional image analysis by calibrating, pre-computing known extents, pixel alignment and storing metadata in a cell lattice structure. The temporal-pixel aligned data can often be analysed faster across space and time dimensions than previous scene based techniques. This allows the AGDC to be flexible in tackling future challenges and improve analysis times on every-increasing data repositories of earth observation.

The AGDC has also been used internationally to allow countries to maintain ecologically sustainable programs and reduce the difficulty curve of utilizing Remote Sensing data.

Background

The AGDC was originally conceived by Geoscience Australia but is now maintained in a partnership between Geoscience Australia, Commonwealth Scientific and Industrial Research Organisation (CSIRO) and National Computational Infrastructure National Facility (Australia) (NCI). This is made possible by the funding from the partnership and a number of organisations such as National Collaborative Research Infrastructure Strategy (NCRIS).

Analysis ready data, ingestion and indexing

The data processed in the cube is made analysis ready before being ingested and indexed into the AGDC. Analysis ready data is pre-processed data that has applied corrections for instrument calibration (gains and offsets), geolocation (spatial alignment) and radiometry (solar illumination, incidence angle, topography, atmospheric interference). The ingestion process manages the translation of datasets into the storage units while maintaining a database index. The data within the storage and index can be accessed via API calls often compiled within code such as Python (programming language).

Example:

s2a_l1c = dc.load(product='s2a_level1c_granule',x=(147.36, 147.41), y=(-35.1, -35.15), measurements=['04','03','02'], output_crs='EPSG:4326', resolution=(-0.00025,0.00025))

Datasets currently stored

• Geoscience Australia Landsat Surface Reflectance (1987 to present)
• Landsat Pixel Quality
• Landsat Fractional Cover
• Landsat NDVI

Datasets that have been piloted

• USGS Landsat Surface Reflectance
• SRTM DEM
• Himawari 8
• MODIS
• Sentinel-2 L1C / S2A
• Australian Gridded Climate Data

Open source

The AGDC code base is situated in GitHub as an open repository. The core code base moved to the Open Data Cube in early 2017 as part of an international collaboration. Whilst the code base is the Open Data Cube, individual cubes exist as their own right such as the AGDC on the National Computational Infrastructure National Facility (Australia) (NCI) using the High-Performance Computing Cluster HPCC. The core code can be installed on personal computers or public computers (using git) and has many unit tests.

Documentation for the code base exists on Read the Docs.

Challenges of the AGDC

The AGDC is designed to meet nationally significant challenges such as the following.

• Sustainability
• Environment
• Water resource management
• Disaster assist
• Policy development
• Community planning
• Forest preservation
• Carbon measurement

International awards

The AGDC won the 2016 Content Platform of the Year award from Geospatial World Forum.