Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tests] Skip XslCompiledTransformApi tests on iOS, tvOS, and when testing native AOT #75730

Merged
merged 6 commits into from
Sep 23, 2022

Conversation

mdh1418
Copy link
Member

@mdh1418 mdh1418 commented Sep 15, 2022

Fixes #75699

In effort to consolidate System.Private.Xml Test projects, the test suite exclusions within https://github.com/dotnet/runtime/blob/main/src/libraries/tests.proj weren't being honored because the suites no longer existed.

Instead, to skip the tests, [ConditionalClass(typeof(PlatformDetection), nameof(PlatformDetection.IsReflectionEmitSupported))] is being added at the class level to skip tests where Reflection Emit is needed.

This PR skips tests within XslCompiledTransformAp and XmlSerializerTests

@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost assigned mdh1418 Sep 15, 2022
@mdh1418
Copy link
Member Author

mdh1418 commented Sep 15, 2022

/azp run runtime-extra-platforms

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Comment on lines 624 to 625
<ItemGroup Condition="(('$(TargetOS)' != 'tvOS' and '$(TargetOS)' != 'iOS') or '$(RunDisablediOSTests)' == 'true') and
('$(TestNativeAot)' != 'true' or '$(RunDisabledNativeAotTests)' == 'true')" >
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be better to put [ConditionalClass(typeof(PlatformDetection), nameof(PlatformDetection.IsReflectionEmitSupported))] on the individual test classes in these files. It's more consistent with how we do things and more futureproof.

@mdh1418
Copy link
Member Author

mdh1418 commented Sep 16, 2022

/azp run runtime-extra-platforms

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Member

@stephentoub stephentoub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if it fixes the problem.

@filipnavara
Copy link
Member

filipnavara commented Sep 19, 2022

It fixes half of the problem. Other half of the NativeAOT tests still fail and need similar treatment.

@mdh1418 mdh1418 marked this pull request as ready for review September 19, 2022 16:30
@mdh1418
Copy link
Member Author

mdh1418 commented Sep 19, 2022

This PR was initially only targeting XslCompiledTransformApi tests, so I didn't have a look at all the other test suites that were removed and where the respective TestExclusions weren't honored, but I'll look at the failing XmlSerializerTests tests. Would it make sense to make PRs on a test suite by test suite basis, or trying to tackle them all in this one PR?

@filipnavara
Copy link
Member

I don't know what triggered it but my totally unrelated PR started seeing the same errors on the NativeAOT pipelines. In fact I started getting them locally for the main branch when building the NativeAOT tests. That's why I started watching this PR since it resolves a sizeable amount of the failures. I appreciate that you started looking into it even if it was not for the same underlying reason.

Btw, you should run the extras pipelines (/azp run runtime-extra-platforms) once you are reasonably sure that the tests from the previous run are now marked correctly.

@MichalStrehovsky
Copy link
Member

don't know what triggered it but my totally unrelated PR started seeing the same errors on the NativeAOT pipelines

The test projects (mentioned in the diffs to src/libraries/tests.proj in this PR) got deleted and folded into another project that is not excluded. So tests we weren't even compiling before started failing.

@filipnavara
Copy link
Member

The test projects (mentioned in the diffs to src/libraries/tests.proj in this PR) got deleted and folded into another project that is not excluded. So tests we weren't even compiling before started failing.

Ah, that explains it. I can understand that it was missed as part of a PR if the extras pipelines were not run but I am quite surprised that there was no issue filed for it based on the rolling builds.

@stephentoub
Copy link
Member

#75699 (comment)

@filipnavara
Copy link
Member

#75699 (comment)

Missed that one (or rather it was not filed yet when I first hit the issue). Thanks.

@mdh1418 mdh1418 force-pushed the properly_skip_xslcompiledtransform branch from 00e9588 to 966cfea Compare September 20, 2022 04:54
@mdh1418
Copy link
Member Author

mdh1418 commented Sep 20, 2022

/azp run runtime-extra-platforms

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mdh1418
Copy link
Member Author

mdh1418 commented Sep 20, 2022

It fixes half of the problem. Other half of the NativeAOT tests still fail and need similar treatment.

I looked at the test failures more closely, and it doesn't look obviously System.Reflection.Emit to me.
Seeing errors like System.InvalidOperationException : There is an error in XML document (2, 14).

I looked at the comment in tests.proj and it looks like System.XmlSerializer is just not supported on NativeAOT right now.

I also noticed some test failures in two more suites that were removed System.Xml.Xsl.XslCompiledTransformApi.Tests.csproj however there doesn't seem to be a test suite skip in tests.proj, so not sure how these were skipped in the past. They were failing with

System.Xml.XmlSchemaValidatorApiTests.VerifyException : Error while loading assembly
   at System.Xml.XmlSchemaValidatorApiTests.ExceptionVerifier..ctor(String assemblyName, ExceptionVerificationFlags flags, ITestOutputHelper output) + 0x5fd
   at System.Xml.XmlSchemaValidatorApiTests.TCInitialize..ctor(ITestOutputHelper output) + 0x48
   at System.Private.Xml.Tests!<BaseAddress>+0x12ad20d
   at System.Reflection.DynamicInvokeInfo.Invoke(Object thisPtr, IntPtr methodToCall, Object[] parameters, BinderBundle binderBundle, Boolean wrapInTargetInvocationException) + 0x150

@filipnavara
Copy link
Member

I looked at the test failures more closely, and it doesn't look obviously System.Reflection.Emit to me.
Seeing errors like System.InvalidOperationException : There is an error in XML document (2, 14).

It's a bit misleading but if you look closely at the stack trace the inner exception is still PlatformNotSupportedException due to unsupported Reflection Emit.

Tests fail from trimming and MakeGeneric, and XmlSerializer is not supported with NativeAOT
@mdh1418 mdh1418 force-pushed the properly_skip_xslcompiledtransform branch from 966cfea to 103d596 Compare September 21, 2022 17:11
@mdh1418
Copy link
Member Author

mdh1418 commented Sep 21, 2022

/azp run runtime-extra-platforms

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@jkotas
Copy link
Member

jkotas commented Sep 23, 2022

This is not fixing the whole problem, but it is still an improvement. Merging.

@jkotas
Copy link
Member

jkotas commented Sep 23, 2022

Actually, I do not think that the condition on the Schema and XmlSerializer test is right (it should not be ReflectionEmit). The Xslt one is good, so let's merge that one at least.

@jkotas jkotas merged commit 74c589f into dotnet:main Sep 23, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Oct 24, 2022
@mdh1418 mdh1418 deleted the properly_skip_xslcompiledtransform branch October 28, 2022 02:18
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[iOS] XslCompiledTransform tests failing on devices
5 participants