ESA > Join & Share > Technology Projects > HiProGen Project

HiProGen Project

Project Title   High level information Product Generation and formatting for specific applications
Project Acronym   HiProGen
Contractor(s)   Infoterra Ltd., VTT


Project Objectives
          How it Works


Objectives   Top

The High level Product Generation (HiProGen) project aims to demonstrate a system that can automatically convert low level Earth Observation data to geocoded, geophysical composites and wide area mosaic products.  The project has objectives for both the demonstration system and for the demonstration products.  The product objectives are as follows:

  Image generation of five* demonstration terrestrial products as weekly and monthly global composites
    (*Albedo, Global Vegetation Index, fraction of Photosynthetically Active Radiation, Leaf Area Index, Net Primary Productivity);
  Image generation of a Global Algal Index demonstration marine product as weekly and monthly global composites;
  Image generation of demonstration ASAR mosaic products with regional and global coverage;
  Image creation of global scale products with a spatial resolution similar to that of the measuring instrument;
  Image output of demonstration products to various common EO formats and projections.

The principle system objectives are to demonstrate:

  Image automatic generation of Level 3 products from lower level Earth Observation data
    (specifically MERIS Level 1 and 2, ASAR Level 1 Wide and Global Monitoring and Level 1 ATSR-2 / AATSR);
  Image a flexible architecture which allows easy extension of the system to generate new and improved Level 3 products;
  Image an expandable and portable system which can be distributed across several platforms to improve performance;
  Image WWW capable user interfaces for product / production search and order.


Context   Top


Earth Observation (EO) data in its raw, low level form is a stream of numbers which have little meaning to the vast majority of potential users.  As this low level data is processed to higher levels, it becomes more accessible to and more meaningful to a wider range of users.  The data processing levels are classified as follows:

  Image Level 0 - raw instrument data;
  Image Level 1A - Level 0 data, which may have been reformatted or transformed reversibly, located to a coordinate system, and packaged with needed ancillary and engineering data;
  Image Level 1B - Radiometrically corrected and calibrated data in physical units at full instrument resolution
  Image Level 2 - Retrieved environmental variables (e.g., ocean wave height, soil moisture, ice concentration) at the same location and similar resolution as the Level 1 source data.
  Image Level 3 - Data or retrieved environmental variables that have been spatially and/or temporally resampled (i.e., derived from Level 1 or Level 2 data products). Such resampling may include averaging and compositing;
  Image Level 4 - Model output and/or variables derived from lower level data which are not directly measured by the instruments.  For example, new variables based upon a time series of Level 2 or Level 3 data.


Level 1 and Level 2 data products have several limitations:

  Image they provide a single, instantaneous set of measurements along a swath of ground below the satellite;
  Image the data are delivered in ESA / instrument specific formats;
  Image sample values are georeferenced but are not available on a standard geographic coordinate grid (geocoded).

With HiProGen, ESA is exploring the operational production of global scale, geocoded products available in a variety of data formats and map projections.  The combination of multiple data acquisitions, over weekly and monthly time periods, results in composite products with averaged values and a reduced loss of coverage due to cloud cover.  Traditionally, Level 3 products have used large spatial sampling areas or bins to accumulate input data.  By resampling data to a common grid for each type of input data, HiProGen will generate Level 3 products with far better spatial resolutions than are conventionally available.


Architecture   Top

The HiProGen system is composed of seven components (see diagram below), each of which can be deployed on a dedicated workstation connected to the rest of the system across a LAN.  This modularity allows the system to be distributed and expanded to meet increased processing requirements - an important consideration when each global composite product can reach 25 GB in size!

The system also makes use of standard formats and protocols for its interfaces: XML for configuration and processing instruction files and HTML / Java to enable any computer with a Web browser to access the user interfaces.




How it Works   Top

A number of Use Cases are supported by the HiProGen demonstrator: data search, image browse; data ordering; etc. however the principal use of the system is for the automatic generation of Level 3 products.  The activity diagram below shows how the core sub-systems interact to process input Level 1/2 data.  The Data Ingestor regularly polls the input directories for new low level data.   Once a new data granule is discovered, the Data Ingestor extracts relevant metadata, creates browse images and then passes the information to the File Server.  Following archiving of the data granule by the File Server, the database primary key for the data is returned.  The Data Ingestor uses this primary key to populate an instructions template before transfering the instructions file to a staging directory.


The Process Controller regularly polls the instructions staging directories for new instructions files.  Once a new instructions file is discovered, the Process Controller determines the tiles intersected by the input data.  For every tile, the instructions list is converted to a list of processes.  As each list of processes is executed, a Level 3 product tile is created / updated before finally being archived by the File Server.


Output   Top


HiProGen will provide ESA with:

  Image the ability to generate examples of high resolution Level 3 products for internal / external evaluation;
  Image a demonstration system which can be used to explore the throughput, scaling and storage requirements of an operational product generation system;
  Image the capability to generate new Level 3 demonstration products by the addition of further processing chains.

Products would be available in the following data formats: HDF-EOS 4; ENVI-BIL; ENVI-BSQ; GeoTIFF; ERDAS LAN and ER-Mapper.

System stress testing of the HiProGen system has been carried out with over 160 orbital segments of MERIS Level 1 RR data for the land products and over 140 passes of Level 2 RR data for the marine Global Algal Index (GAI) products. Processing of a single, 80°N to 70°S orbital segment takes under 100 minutes on a dual 2.4 GHz Xeon PC.  In other words, the HiProGen demonstrator can perform geophysical processing of input MERIS data and composite into the output product faster than input data is acquired. 

One of the requirements for the HiProGen demonstrator was that it should generate Level 3 products at the resolution of the parent instrument.  For MERIS, this means output products with nominal sample sizes as fine as 300m.  In order to cope with the high data volumes of the Level 3 products, HiProGen outputs are automatically created as square latlon tiles.

The use of tiling is highlighted in the psudo-colour image of Global Algal Index (GAI) Case 1 waters shown to the right.  This coverage of the central Mediterranean was generated using algal_1 MERIS RR data for July 2003, incorporating a single quarter scene of MERIS FR data.  Output sample size is 300m x 300m.


The pseudo-colour image above shows the global land surface albedo for April 2003 in a Plate Carree (equi-rectangular, non-projected) map with an equatorial spatial resolution of 1.2km.
The cloud masking algorithm excludes data with very high reflectance.  Note the effect on ice covered areas and the Sahara desert.


Contributors to this page: andreadv .

Page last modified on Tuesday 14 of December 2010 13:55:37 CET by andreadv.