Noisy many "Tests skipped" when only some of the collections support the Features API, timeStamps

[jerstlouis]

In light of the resolution of the big Collections discussion ( opengeospatial/ogcapi-common#140 ), where some of the collections of a dataset could be accessible as Features, some as Coverages, and some as both, there should probably be an "early skip" without the noise for Collections not offering "items".
e.g. if I set a max number of collections to test to 5, the first 3 are raster collections without Features that cause "Skipping test" noise in the results.
In our OpenAPI definition this is actually properly reflected, but the test still gets noisy about these.

Similar comment for Feature collections without a timeStamp property -- not all collections would be temporal or expose times...

[dstenger]

Thank you for the input.

Can you provide a test service which is causing much noise/ is long running?

[jerstlouis]

@dstenger http://maps.ecere.com/ogcapi with noOfCollections = 5.
It will show Skip = 45 with skip messages like:

org.testng.SkipException: Property timeStamp is not set in collection items 'NaturalEarth:cultural:ne_10m_admin_0_antarctic_claims'
org.testng.SkipException: Could not find collection with id NaturalEarth:raster:HYP_HR_SR_OB_DR in the OpenAPI document
org.testng.SkipException: Could not find url for collection with id NaturalEarth:raster:HYP_HR_SR_OB_DR supporting GeoJson (type application/geo+json)
org.testng.SkipException: Could not find a response for collection with id NaturalEarth:raster:HYP_HR_SR_OB_DR

None of the collections have time stamps, and the raster collections don't support OGC API - Features.

[ghobona]

So in pseudo code, it would be.

Retrieve /collections

For each ID in collections:
     Retrieve /collections/ID
     If response of /collections/ID has JSON Object with "rel" : "items":
           Add to list of collections to test with this ETS

Run compliance test on list of collections to test with this ETS

[cportele]

@ghobona - Since this is the ETS for Features and not for, e.g., Records, also inspect the itemType, if there is an items link. If the property is present and has a value that is not "feature", ignore that collection, too.

[dstenger]

@keshav-nangare Can you please update the implementation regarding the proposed solutions?

[keshavnangare]

Added check to collect collection if the itemType is feature or it is absent but it must have "rel" : "items". Please find the changes in #138 PR.

[jerstlouis]

About FeatureCollections without timeStamp, could this "test skipped" appear only once maybe, if none of the collections have a "timeStamp" property? (And not appear at all if at least one collection had a timeStamp and the test was executed for that one)

[ghobona]

2021-02-18 meeting

@ghobona discussed the "test skipped" suggestion with @dstenger and @keshav-nangare , and there was agreement that there is need to maintain the current approach of testing the timeStamp property on each collection.

[jerstlouis]

@ghobona To clarify, my suggestion was to test the timeStamp property on all collections that do have it, but report only once that tests were skipped on the n collections that don't have it, so that you don't end up with 960 "test skipped" if you test 1000 collections and only 40 have a timeStamp.

[dstenger]

Unfortunately, this issue was closed as the pull request connected to this issue was merged.

@jerstlouis Can you please open a new issue if the skip behavior of the timeStamp tests shall be improved?

[jerstlouis]

@dstenger done: #143
Thanks.