ESA > Join & Share > Forums > LTDP SAFE > Externalisation of representing information



Show posts:
Jump to forum:

Externalisation of representing information

According the SRR outcomes, it was decided to perform a trade-off to evaluate if it is really worth to extract the representation information out of the SAFE products or if, on the contrary, it is advisable to keep it inside.

The attached document “Externalisation of Representation Information Trade-off” (PDGS-SAFE-GMV-TN-12-0071) provides the analysis and conclusions reached about this topic.

We can conclude that the externalization of the representation information can be implemented but under certain conditions:
1) it is assumed more implication on the archive responsible side is needed
2) it is assumed that archive infrastructure has to ensure the following:
a. The representation files are properly stored/organized in the same archive.
b. The representation files are accessible for the actors/tools in charge of creating/updating the SAFE products.

Current topic discussion and the analysis included in the attached document are devoted to reach a consensus before the PDR-C because the final solution may imply a change in the format design that has to be considered for the SAFE Core Specification update.

All your comments will be appreciated.

Adrián Sanz (GMV)
LTDP SAFE Project Manager

Re: Externalisation of representing information

Detailed feedback has been provided by Stephan Zinke (EUMETSAT Panel member) supporting the conclusions provided by GMV, but taking into account some necessary recommendations.

The attached memorandum provides additional elements for discussion that has to be considered in this topic.

Re: Externalisation of representing information

Hi Stephane,

Thanks for your comments. You can find my answers in the attached doc.
I just copy below one of the answers, since it refers to missing info in the trade-off that can be of interest for the others.


Stephan's Comment
Section 3.3, p14: It is not really clear what this section is about, and I am a little confused by the use of “XML Schemas” and “XML schemas”. A distinction should be made between “general” XML Schemas and “product-type specific” schemas (SDF / DFDL), if that was the purpose of this section.

GMV's Answer
Oops, my fault about the upper and lower case, for me they should all be the same (let’s say XML Schemas in upper case). Sorry for that.
I agree with you that this section needs more details, since now it doesn’t explain how some of the representation information is preserved (as the DFDL schemas). I give you here the idea that we have in mind: we plan to widen the SAFE format to be able to describe not only EO products, but other types of files as well, such as auxiliary products or representation schemas. We propose to have a hierarchy of abstract classes of SAFE providing a set of metadata applicable for each node and for the lower levels. For example, the hierarchy for an EO Product will be: Abstract SAFE -> SAFE DATA -> SAFE LTDP -> SAFE EO Product, and the hierarchy for a representation schema will be: Abstract SAFE -> SAFE Representation. In this way, the representation schemas will be stored in the archive as well in SAFE format hence they will be preserved as well. Inside a SAFE EO product, the references to the representation information will be in fact references to other files in SAFE format. I hope this throws a bit of light on it.

Re: Re: Externalisation of representing information

Dear Hector,
thank you very much for your answers. Please find below some more comments. For all the rest I'm fine with the answers supplied, i.e. to update the document where necessary and/or give explanations like you did in your response document.

>First sentence will be changed: “in a way that allows to access to its location”. I >agree with you that we have to consider the case of an intermediate API in >charge of accessing the file. I don’t see a big difference in the approach anyway. >What do you think? IMO, in that case the reference should be treated as a >logical reference by the API (logical path + filename) instead of a physical >reference. The API should know how to interpret this reference.

An explanation in the document(s) would be sufficient. The difference in the architecture is a decoupling by using some intermediate layer.

>Of course, I assume that read-only permissions are possible, but from where >and for which archives? Can we ensure that all the archives will share the same >security policy? I was assuming that the archive could be accessed from an >internal LAN or similar (e.g., HARM was going to be able to access), but that >outside the firewalls of the archive we couldn’t easily grant permissions (not >because of technical reasons, but due to the different security policies of each >archive).

There seems to be a stress here on "all the archives". I am not sure if this is necessary. Different archives can be implemented differently. What they would share, though, is that they store data in SAFE formats and that the SAFE requirements apply. IMO, the archives themselves are a little transparent / irrelevant for SAFE, as long as they fulfill all the requirements.
It is as well the question, "who" is going to access an archive (with or without the help of the toolset). I believe this can be realized by security processes and procedures and roles.

>I am not sure to understand the link here. Even for read-only access some >credentials may be needed. The security in this case is just for avoiding to write >the credentials (even encrypted) in the SAFE product. I was assuming that if this >could be achieved, it would be better for everybody.

OK, I understand your point now.

>Ok, well, I didn’t want to enter into XML details. Anyway we can provide a simple >example in a standard context, i.e., in a situation where the XML schema is been >accessed by the XML instances. Imagine that a new version of a XML Schema is >created (using the same namespace, as recommended in point 1 of section 4.1) >and the new version simply overwrites the old version. ...

OK, I understand your rationale now. However, it should be forbidden to do what you explaint. I.e., each file (for each version) should have a unique name, preferably inlcluding the version info.
Additionally, one can take advantage of the version-info (attribute) inside any XML file (if defined).
For me, personally, I would want all file versions in one place, and could distinguish them by the name...

>It is just a recommendation according to the information that I have.
>This configuration is something that will change from time to time ...

In this case, IMO, it makes sense as well to store the tools themselves (as e.g. binary executables)

Re: Externalisation of representing information

Dear Adrian and Hector,

This trade-of has generated a great number of discussions in our team.

Even if the current version of the SAFE format makes the rep info preservation easier in a long-term basis, the workload of transcribing all the products upon potential future changes in the rep language seems to be a good reason for externalizing the info and reducing the impact on the archives. However a bigger implication of the responsible of the archives should be requested and the risk of having the information out of the product itself must be minimized by the different centres (critical points: network security rules, firewalls, system access, etc.).

In any case, even at this stage, the solution for this trade-off seems to be complex and we think it should be better evaluated if the costs and resources needed to implement this change now are less than the ones needed for a possible transcription in the future.

The rest of our questions are fully covered by Stephan’s questions and your answers. Just a remark regarding section 4.1. pt 9: We would also prefer to have all file versions of the rep schema in a same location and distinguish them by the name (updated version number) instead of having them in different directories.


Re: Externalisation of representing information

The following comments have been done offline by Diego Lozano (Deimos) after the PDR-C collocation meeting:

One of the advantages of the externalizing the representation is that the representation language can be changed in the future. In the tradeoff Externalisation of rep. information Ref. PDGS-SAFE-GMV-TN-12/0071, the problem of the versioning of the schemas has been analysed. Linking the products to a fixed version, how is the procedure going to update the language of the representation?

- Replace the previous SAFE representation package with the new language for all the current versions.

- Create a new version of the SAFE Representation packages, so then :

  • ) do we need to transcribe the products only change the manifest for linking the new version?
  • ) Can be the links to the representation version independent? If so, new versions of a representation package shall be backwards compatible.

Show posts:
Jump to forum: