Loading...
 
ESA > Join & Share > Forums > HMA Forum > HMA Online Data Access Time Series Support

HMA Forum

Help

Show posts:
Jump to forum:

HMA Online Data Access Time Series Support

The support of time series retrieval on EO datasets was subject to discussion in HMA-FO Online Data Access task, and we would like to continue this discussion based on the progress made. Please find attached our current ideas on this topic, reflecting examples of timed EO datasets and expectations from a WCS with temporal selection support.

Re: HMA Online Data Access Time Series Support

Dear Stephan,

thanks for these examples. We carefully studied your interesting slides.
Some of our questions have already been addressed by EUMATSAT. However, her are some more.

Slide 2:
The examples for time-series are generally intuitive but contain some phrasing which is unclear in the context of online access to time series.

*) what do you mean by "Aggregation of L2 products to composites" ?
- shall the WCS do the aggregation on the fly?
- or shall coverages be offered which have been prepared somewhere else

*) what do you mean by "Assimilation of L2 products to full coverages"
- again, the same two questions as above
- what is the difference between "Assimilation" and the above "Aggregation"?

Slide 3:
Certainly, CRSes with time would be nice (but probably complex and tricky to handle). Could you, by any chance, provide an example of such a CRS?

We also agree that WCS should support existing formats, and we know that several WCS format encoding extensions i.e. for netCDF, GeoTIFF, JPEG2009 are already under preparation.

We can understand your needs for more complex time queries. These examples are certainly helpful and should be considered. However, such request scenarios are already possible. Just not in a single step (see current WCS EO AP approach below).

The future expectations shed some light on future use cases, although some of them (e.g. iceberg/ship motion monitoring) are very complex.
Besides our understanding for those use cases and your wish to solve such complicated scenarios with a single request, we still don't understand how a 3D-coverage could solve these issues.

Slide 4:
Could you provide us with some small, real world examples of how the offered and the retrieved data you expect should look like? We would like to understand how the results from such requests shall be internally organized and what file format(s) you expect.
The graphics here, however, seem to imply that you always expect one or multiple stacks of coverages and not 3D-coverages as defined by OGC.

The current WCS EO AP approach:
The user submits a DescribeEOCoverageSet request providing the eoId of a dataset series, and specifying the spatial and temporal limits (i.e. AOI and TOI) as constrains. He will receive CoverageDescriptions for all datasets contained in the eoId's dataset series and matching the provided limits.
Yes, now the burden is on the user/client side to evaluate all the metadata and then issue multiple GetCoverage requests.
With this method you can potentially get anything you want, in any interval. This allows to build highly specialized clients fulfilling exactly predefined requirements, by providing the respective analyzing tools to evaluate the received CoverageDescriptions, in one or the other way.

We are well aware that there are, as always, advantages and disadvantages on both sides (server and client), but such will exist for any solution. Maybe we can discuss implications such as server performance, client design, etc. at a later stage. For the time beeing we would like to gain a better understanding of your needs.
ciao
Christian


Re: Re: HMA Online Data Access Time Series Support

Dear Christian,

Let me try to answer your questions:

Slide 2: Here the aggregated/assimilated data set examples are systematically generated by the underlying payload ground segment. This usually requires expert knowledge, because aggregation/assimilation cannot be done through standard statistical methods. When aggregating atmospheric spectrometer data, e.g. the sensing geometry has impact on data quality, which should be taking into account when aggregating overlapping pixels.

Slide 3: Yet the time series we produce have the temporal granularity and reference as provided by the mission (sensor or processing scenario), without reference to a 3/4-D-CRS. In internet we find a lot of descriptions how netCDF can be used for time series, incl. within OGC, e.g. you may have a look at http://external.opengeospatial.org/twiki_public/bin/view/NREwg/SWECommonNetCDFTutorial , or directly ask Alexandre Robin for standards. It would be interesting to have a survey on data formats/profiles with time series support for raster EO data. By origin and within OGC/SWE, time series often refer to point data.

Slide 4: I attached a real example, a 2-D-plot with time and latitude (showing the long-term evolution of total atmosphere ozone depending on the latitude), at a constant longitude or as mean of all latitudes. I hope this helps for understanding. There are other examples that also include the height above earth surface as additional dimension, where time series show the seasonal ozone hole generation, motion and dissolution as function of latitude and height. For this kind of results it is of course helpful, if the WCS supports standard statistical methods e.g. to calculate means on dimensions which are subsampled/grouped or hidden.

We do not see a difference between a "stack of 2-D coverages" and a "3-D coverage". As for spatial dimensions, the time dimension also needs a gridding, so in any case a 3-D coverage is raster data where each pixel has a reference in time and space.

Thank you for the explanation of the current WCS EO AP approach. We agree that implications of this intermediate solution need to be analysed and discussed at a later stage. This will hopefully lead to a next WCS version where the time series can be computed at server side.

Best regards, Stephan


Re: Re: Re: HMA Online Data Access Time Series Support


Dear Stephan,

thanks for your answers and your example.

We fully agree, and so was our understanding, that in most cases the aggregation/assimilation of datasets requires expert knowledge. This, however implies that the L2 composites/full coverages, you're mentioning, are pre-arranged and can therefore also be addressed as single coverages (or as a DatasetSeries, if there are more of the same). So this case is fully covered by the WCS EO-AP.

Let me jump ahead and come back later - you state:
> We do not see a difference between a "stack of 2-D coverages" and a "3-D coverage". As for
> spatial dimensions, the time dimension also needs a gridding, so in any case a 3-D coverage is
> raster data where each pixel has a reference in time and space.

You are right that using a 3D-CRS for a GridCoverage the "time dimension also needs a "gridding" but it is exactly this gridding which might cause unwanted side-effects. While the grid of 2D raster data is usually regular spaced, this will not necessarily be the case for the time-axis. Imaging that an area will be covered by e.g. SeaWiFS data only every 2-3 days. So you will need at least a gridding of the time axis at a day's interval. Therefore you will end up with some dates having no data. If you want to work with more exact timing on the time-axis e.g. seconds or even milliseconds, to match the exact start-time of the satellite imagery, then you will end up with a lot of empty space between each observation. Furthermore, if we consider the timing strictly (e.g. in milliseconds) and apply your requirement (where each pixel has a reference in time and space) then we end up with satellite images being split in the time direction (because of the duration of the data take some lines/pixels will at time A while others are at time B or C).

Of course, one could also use an index on the time-axis to get rid of the empty space, but this will not allow to extract data by time subsetting directly. The client application would need to translate the applied index into the time-values received via DescribeCoverage/DescribeEOCoverageSet or the description of the DomainSet gets very big. Not very elegant either.
While these considerations are based on the variability found in satellite remote sensing, a time-series of measurements from a drifting buoy or any other sensor, where measurements are made at regular intervals, will not suffer from these limits. These coverages are modeled e.g. as PointCoverages. However, there's no definition nor experience for mixing GridCoverage (for spatial axis) and PointCoverage (for time axis).

So the following question with a 3D-CRS remains: How shall the empty spaces on the time-axis be handled? What coverages shall be delivered if there is no data available at the requested time? Empty coverages containing only Nil-values or an Error-Message? How shall a long-lasting data take (e.g. a full stripe) be represented on the time-axis? Only with the start-time or using the real time of the take of the respective pixel?
We have been discussing these issues a lot and agreed that a 3D-CRS (or even 4/5D-CRS) would certainly be nice and such CRSs will probably become available in the future.

However, for the time being, we decided to choose another solution to reach our target i.e. to extract sub datasets from any time-series. This solution doesn't use the time as an axis of the CRS, but rather uses the time information available from its metadata to subset the offered series during a DescribeEOCoverageSet. Such requests will return the IDs of the available datasets, which can then be accessed. This solution has the positive side effect that there will be no limitation in the data/file formats to be used (as long as the corresponding metadata record is available).

However, it has to be mentioned that, at the current state of the WCS EO-AP, only a single time trim (i.e. a point in time or one time interval) is foreseen in the request. The request you mention on your slides (regular intervals, calendar intervals, irregular intervals) are possible by issuing multiple requests. But again the burden would be on the client application side. The price one have to pay is that sometimes multiple request are necessary. On the other hand this allows to create highly specialized clients optimized for a certain use case/application.

Regarding your examples, you have chosen aggregated/assimilated dataseries (i.e. L3-products), which usually do not suffer from irregularities on the time-axis as discussed above, and hence will not be touched by the problems associated with the time-axis in 3/4D-CRS.
However, for the WCS EO-AP we had to keep a wide view and we tried to cover the various aspects. We also had to consider other requirements like e.g. limit the size of the GetCapabilities document, not to invent new requests (if not absolutely necessary), etc. And last but not least, there are currently no 3/4/5D-CRSs available and we had to keep in schedule.

Regarding your statements: *) "it is of course helpful, if the WCS supports standard statistical methods e.g. to calculate means on dimensions which are subsampled/grouped or hidden" --> this is not part of the regular WCS functionality, but would require a WCPS (or WPS). *) "This will hopefully lead to a next WCS version where the time series can be computed at server side." --> I assume this statement means: a single request will return a series of datasets (in one or multiple files). As mentioned above, such behavior could alrady be implemented by a client performing the required requests transparently to the user. The WCS EO-AP has been submitted to OGC. We tried to create a lean and easy to use/understand AP covering as many requirements as possible. The upcoming discussions at OGC's WCS.SWG and at the TC meeting (Bonn) will show if we succeeded. Further discussions e.g. also in the upcoming OWS-8 initiative, will provide the opportunity to add/modify/extend certain behavior currently suggested by the WCS EO-AP. ciao Christian


Re: Re: Re: Re: HMA Online Data Access Time Series Support

Christian,

thanks a lot for your very comprehensive comment.

Just a short addition on the 3D-CRS and time-gridding aspect: As discussed, EO time series are typical for higher level products, where time and space gaps between single scenes are systematically filled by aggregation/assimilation methods, leading to stored EO products without sensing-related gaps. E.g. for atmospheric data, complex chemical and wind transport models are used for this assimilation. So the "no value"-case should still be the exception. Lower level EO collections may of course not be well suitable for a representation as time serie, because a good time gridding may be difficult to find.



Re: HMA Online Data Access Time Series Support

I'm in favour of your suggestion to enhance the time based functions, since this would be very much appreciated by EUMETSAT.
A few questions regarding the slides:

  • "Data formats with time dimension support (combination of geo and video)": There are already data formats (in use) that include the time dimension (e.g. netCDF). Having the data displayed as a video seems to me more like a visualization operation/tool, or more the domain of WMS.

  • "GetCapabilities returning also time resolution and time coverage of dataset": I don't understand why this has to be in GetCapabilities instead of DescribeCoverage.

  • "Extension: moving target support (e.g. iceberg/ship motion monitoring, ocean currents monitoring) via 3D vector track for spatio-temporal selection": I don't understand this point.

Re: Re: HMA Online Data Access Time Series Support

Some answers to the very good EUMETSAT comments:

> * "Data formats with time dimension support (combination of geo and video)": There are already data formats (in use) that include the time dimension (e.g. netCDF). Having the data displayed as a video seems to me more like a visualization operation/tool, or more the domain of WMS.

Right. Current WCS developments should allow use of existing tools and formats as far as possible.

> * "GetCapabilities returning also time resolution and time coverage of dataset": I don't understand why this has to be in GetCapabilities instead of DescribeCoverage.

Of course, DescribeCoverage is the right request to return this kind of information, rather than GetCapabilities.

> * "Extension: moving target support (e.g. iceberg/ship motion monitoring, ocean currents monitoring) via 3D vector track for spatio-temporal selection": I don't understand this point.

The common understanding of time series is currently based on the assumption, that the spatial selection for subsetting (in the GetCoverage request) is constant in time. What if an application looks at e.g. the temperature evolution of the gulf stream or the evolution of an oil spill on this ocean current. The spatial selection could become a target moving in time, which could be described with a 2D geoobject and a vector for motion of the geoobject in time. In the resulting subsetted time serie, the spatial coverage would then move from one time unit to the next time unit...

Again - I see this feature as a future option and not in the current focus. We first need a WCS which allows efficient retrieval of time series with fix spatial subsetting.



Re: HMA Online Data Access Time Series Support

Thank you for the detailed list of use cases provided! I would like to add an additional use case:
Let's assume we are able to estimate the atmospheric phase screen for SAR products. Assume as well that the estimation can be done for a set of locations where we can exploit additional data. If we need to look at the values along a time axis, how such set of values could be served via a WCS?



Show posts:
Jump to forum: