ESA > Join & Share > Technology Projects > DRPF project

DRPF project

Introduction and Objectives

The main objective of the Distributed Raster Processing Framework (DRPF) project is to design and implement a generic distributed parallel processing framework specifically targeted at processing EO raster images in large scale distributed environments clusters of computing nodes, grids, clouds, etc.). Upon ESA request, the project scope has been replaced, and targeted to the implementation of an Open Source scalable EO Catalogue service based on Mongo database technology.

Expected Results

SOROCA will provide a highly scalable Open Source Earth Observation (EO) catalogue, developed on European Space Agency (ESA) request to speed up and horizontally scale up, fast growing Earth Observation image catalogues.
These catalogues usually store the metadata of remotely sensed images in PostgreSQL database, extended with PostGIS; this solution (relational SQL database) does not offer enough possibility for horizontal scaling (except e.g. using the STADO extension, which nevertheless is not sufficient for our purpose). This is actual at beginning of the project at least, since a new issued generation of PostgreSQL can be strong rival.
SOROCA will provide a new solution based on the MongoDB (no relational) database, as a highly scalable and reliable storage solution, and the Pycsw server as a skeleton for the new catalogue server (SOROCA), developed in conformance with the OpenSearch standard 10-032r3.
The Mongo database does not include a full satisfactory spatial functionality, and due to this reason it needs to be extended. Data in the SOROCA catalogue are divided into blocks of the same size and distributed to several nodes. During the query execution, all nodes including relevant data will be used in parallel.
Catalogue interfaces manage the access to metadata storage, i.e. they search for records based on a client query, and return results in RDF, ATOM, KML, generic XML or WKT format. Query results describe the real data (images) and their source, providing the link to the actual repository for download.

Project Schedule

KOJan 2012ESRINKick-Off 
M1T0+3MISSSystem Requirements Review 
M2T0+6MESRINPreliminary Design Review 
M3T0+12MISSCritical Design Review 
M4T0+16MESRINProgress Meeting 
M5T0+19MISSAcceptance Review 
M6T0+20MESRINFinal Presentation


Iguassu (CZ)

Project Deliverables

Contributors to this page: Michele Iapaolo
SSE Operations Team .

Page last modified on Friday 29 of April 2016 11:45:39 CEST by Michele Iapaolo.