ESA > Join & Share > HMA-T > Conformance testing

Conformance testing

The purpose of this page is to provide a reference place where it is possible to share all information and resources useful to the conformance testing process.

For this procedure is considered as reference the Compliance & Interoperability Testing & Evaluation Initiative - CITE environment developed by Open Geospatial Consortium (OGC).
OGC Compliance Tests are developed via OGC initiatives typically funded by outside sponsors. The participant, selected to develop a test for a specific OGC implementation specification, first documents an Abstract Test Suite (ATS) to indicate what elements of the implementation specification will be tested. After the review by the OGC Technical Committee (TC), from the ATS it is developed an Executable Test Suite (ETS). This script runs within the OGC Compliance Test Engine. In tight conjunction with development of the ETS are built the Reference Implementations (RI), open source implementations of an OGC Web Service, 100% compliant with the associated compliance tests.

Workflow of Compliance Testing process

OGC Web Services Compliance Test Elements

Compliance Test Engine
The test engine adopted by Open Geospatial Consortium is an open source web services compliance engine, Test Evaluation And Measurement (TEAM) engine. It is a cross-platform application that runs in a Java Virtual Machine as a test script interpreter. It executes test scripts written in Compliance Test Language (CTL) to verify that an implementation of an OGC specification complies with the specification.
This engine is an open source software product, available for download from SourceForge at http://sourceforge.net/projects/teamengine.
In the framework of the ERGO project Intecs is updating the TEAM Engine to support SOAP based HMA catalogue interfaces. This modified version can be dowloaded from the following page ERGO TEAM Engine.

Compliance Test Language (CTL)
Compliance Test Language is an XML grammar for documenting and scripting suites of compliance tests for verifying that an implementation of a specification complies with the specification. A suite of CTL files is installed in the compliance test engine, which executes the scripts and determines whether the implementation being tested passes or fails. CTL is based on XSL, which allows the use of a full range of looping and decision constructs and variable declarations. Extension functions can be written in XSL or Java using integrated Java datatypes and standard W3C DOM XML datatypes. Tests can be arranged in a hierarchy, allowing reuse of generic tests and reducing the number of times requests must be made to the service. These advantages should lead to a flexible, intuitive, open and easy-to-maintain test scripts.
A detailed description of the CTL was released as a Discussion paper, OGC document 06-126.

Abstract Test Suite
An ATS, as defined in ISO 19105, is a set of testable assertions specifying all the requirements to be satisfied for conformance, extracted from the specification document. An ATS is described in a conformance clause where abstract test cases, formal basis for deriving executable test cases, are grouped in abstract test modules in a hierarchical structure. An abstract test case shall contain:

  1. test-case identifier;
  2. test purpose(s);
  3. test method (including test-verdict criteria);
  4. reference to specific standard(s);
  5. test types (either a basic test or a capability test).

Test cases are specified independently of any particular test procedure. An ATS may be used to create the respective ETS for a particular test harness.

Executable Test Suite
An ETS consists of CTL Scripts, any Java resource files they may depend on, and related files to be installed online in the engine. Each ETS should be contained in its own subdirectory under the scripts directory of the engine distribution. Some suites may depend on additional components that contain Java extensions to the engine. Components should be contained in a subdirectory under the components directory of the engine distribution. An ETS is hierarchically structured as shown below.


It is possible to download a number of resources like ATS and ETS to the following links (some of them required authentication):

Reference Implementation
RI is a open source implementation of an OGC Web Service which is 100% compliant with the associated compliance tests. OGC has defined a reference implementation as an open source, fully functional implementation of a specification in reference to which other implementations can be evaluated. This is to ensure maximum transparency of its specifications for both vendors and customers.

Test Data
Static dataset, loaded into service implementation to be tested. This data may be necessary for the ETS to test the service implementation.

For further details about the OGC Web Services Compliance Test Program refer to the OGC documents available for download from the following links:

Reference Portal

  • CITE portal: aims to provide a process for testing products complying OpenGIS® Implementation Specifications.
  • OWS4 CITE Wiki: represents the primary place for collaboration for the Compliance and Interoperability Test and Evaluation (CITE) subtask of the OWS-4 testbed initiative.

Compliance Testing Engine (Team Engine) Server

CITE portal is currently able to test software implementations compliance to the following specifications (versions in parentheses):

  • Catalogue Services/Web (CSW 2.0.1)
  • Web Coverage Service (WCS 1.0)
  • Web Map Service (WMS 1.3 & WMS 1.1.1)
  • Web Feature Service (WFS 1.1 & WFS 1.0.0)
  • GeoRSS validation
  • GML Validation
  • Web Map Context (WMC 1.1.0)

It also offers the relatives Reference Interfaces.

The list below describes the compliance tests offers by OWS4 CITE

  • Catalogue Services/Web (CSW 2.0.2 )
  • Sensor Observation Service (SOS 1.0)
  • Sensor Planning Service (SPS 1.0)
  • Web Coverage Service (WCS 1.0.0 & WCS 1.1.1)
  • Web Feature Service (WFS 1.1.0)
  • Web Map Service (WMS 1.3.0)

It is possible to execute a conformance test for the following specification:

  • Earth Observation (EO) extension package for CSW-ebRIM 1.0

Related Resources

  • Test Records?
  • HMA-T Testing Policy, available in the HMA-T Deliverables
  • OGC 08-131r1 The Specification Model — Modular specifications
  • ISO 19105:2000 Geographic Information - Conformance and Testing

Contributors to this page:
Claudio Iacopino .

Page last modified on Monday 23 of March 2009 11:39:05 CET by .

Category: HMA-T