[WIP] Validate manifests against JSON Schema in CI Tests

[r15ch13]

I am implementing manifest validation against the JSON Schema using dlls from Newtonsoft.Json.Schema.
Currently it works on my machine, but it fails on appveyor. (See my build log or the log of this PR)

Could this have anything to do with different powershell or .NET versions?
PS on appveyor:

Major  Minor  Build  Revision
-----  -----  -----  --------
5      0      10586  117

PS on my machine:

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      14393  693

I have no idea why this error Cannot find an overload for "IsValid" and the argument count: "3". pops up.

Any ideas? 😄

[deevus]

Try installing a specific version that works locally.

[r15ch13]

Nope doesn't work either

[rrelmy]

Sorry, I can't help here.
I have no experience with AppVeyor or. NET ;-)

[lukesampson]

Is it possible that the AppVeyor build environment already has an older version of Newtonsoft.Json.Schema installed and it's taking precedence even after you Add-Type the specific version?

Is it worth adding something like this after you run the Add-Type to check the file path of the loaded assembly?

[appdomain]::currentdomain.getassemblies() |? { $_.getname().name -eq 'Newtonsoft.Json.Schema' } |% { $_.codebase }

[r15ch13]

Nice good idea!
https://ci.appveyor.com/project/r15ch13/scoop/build/15-test#L181
Newtonsoft.Json 8.0 is already there 👎

But Newtonsoft.Json.Schema is not installed ...
Edit:
Uuuh i found it in SchemaExtensions.cs#L41
It extends JToken (this JToken source) and i try to call it with 3 instead of 2 parameters because i overlooked the this keyword ... 😢

Other strange things:

• It works on my system
• It works in C#

IList<string> errors;
SchemaExtensions.IsValid(JToken.Parse(strJson), JSchema.Parse(strSchema), out errors);

So i guess PS 5.1 can use static extensions, but 5.0 can't ...

Also using IsValid as an extension doesn't work because $manifest is now of type Newtonsoft.Json.Linq.JObject which doesn't have the IsValid extension... Can i cast it in someway to JToken using Powershell?
RuntimeException: Method invocation failed because [Newtonsoft.Json.Linq.JObject] does not contain a method named 'IsValid'.

$manifest = [Newtonsoft.Json.Linq.JToken]::Parse($json)
$schema = [Newtonsoft.Json.Schema.JSchema]::Parse($schema_json)
$manifest::IsValid($schema, [ref]$validationErrors)

JToken manifest = JToken.Parse(strJson);
JSchema schema = JSchema.Parse(strSchema);
manifest.IsValid(schema, out errors);

Or should i write it as a small C# validation tool (like shim.exe)?

[deevus]

Try calling it using the static class?

[NewtonSoft.Json.Schema.SchemaExtensions]::IsValid($manifest)

[r15ch13]

Already doing that

[r15ch13]

Okay i have written a small c# tool for validating the manifests.
Calling a process for each manifest increased the duration from around 10-20ms to 250-260ms (on an SSD) per test ...

I will try it with a custom library now... 😄

[r15ch13]

Curse you Appveyor!

Again: no idea ...

[lukesampson]

Wow, that's frustrating. Could it be an x86/x64 difference between what AppVeyor is using and what the Scoop.Validator.dll is targeting? Would setting a platform in appveyor.yml make a difference, or making Scoop.Validator.dll match whatever platform the Newtonsoft dlls target?

[rasa]

If you can't get it working, perhaps consider using python for this test, as https://github.com/rasa/scoops/blob/master/validate.py is MIT licensed.

[r15ch13]

Ah nice 👍
Results from trying it out:

• changed validate import to Draft4Validator (because we use the "required" field)
• writing the exception stacktrace to .failed raises an exception 😄 (using Python 3.6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "validate.py", line 51, in <module>
    f.write(trace)
TypeError: a bytes-like object is required, not 'str'

[rasa]

Sorry, I've only tested it with Python 2.7.13. Remove the b in line 50:

with open(failed, 'a+') as f:

and it should work in Python 3

[r15ch13]

Yup that worked. I'm going to redo this in python tomorrow or so. I hope that works on appveyor. 😀

[r15ch13]

Apparently git has desided to destroy all dll files:

warning: LF will be replaced by CRLF in supporting/validator/Newtonsoft.Json.dll.
The file will have its original line endings in your working directory.

Edit: Aah the .gitattributes might be the problem. *.dll is not set.

[r15ch13]

Eureka! It works!

[rasa]

Two thoughts:

1. Was removing UniqueItems intentional? If so, what's the use case for duplicates? Does the code support dups?
2. Requiring version means we can't have pure meta-packages, with just a depends list.

[r15ch13]

1. I have no idea why i removed uniqueItems 😕 Will add it back.
2. Without version the installation won't run at all (see lib/install.ps1#L20)

Added some changes to the schema in #1342