Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make VSTest runner work and enable html log #33915

Merged
merged 1 commit into from
Mar 26, 2020
Merged

Conversation

ViktorHofer
Copy link
Member

@ViktorHofer ViktorHofer commented Mar 21, 2020

The VSTest runner is now supposed to work with the /p:TestRunner=vstest
switch. Currently this is an opt-in but the desire is to get rid of
xunit.console completely and just use VSTest.

An html and trx log example for System.Text.RegularExpressions.Tests: testResults.zip

cc @stephentoub @danmosemsft

The VSTest runner is now supposed to work with the /p:TestRunner=vstest
switch. Currently this is an opt-in but the desire is to get rid of
xunit.console completely and just use VSTest.
@stephentoub
Copy link
Member

Thanks. What does an error like this mean?

Non-serializable data ('System.Object[]') found for 'System.Text.RegularExpressions.Tests.RegexConstructorTests.Ctor'; falling back to single test case.

Did the test run and it just can't report on it, or did it skip it?

@ViktorHofer
Copy link
Member Author

ViktorHofer commented Mar 21, 2020

#30324

Did the test run and it just can't report on it, or did it skip it?

It ran completely but it can't fetch individual data rows, means you won't be able to debug individual data rows in VS Test Explorer. I can only hide those diagnostic messages by disabling all, which might be possible as we only used them for the long running test support for which VSTest might have integrated support.

@jkotas
Copy link
Member

jkotas commented Mar 21, 2020

Are we going to have a workflow in place to make changes in the VSTest runner that are required for not-yet-released .NET Core versions?

For example, let's say that that we add a new platform and this starts failing. How we are going to make progress?

I am asking about this because of I am stuck on a similar problem with BenchmarkDotNet in #33694.

@ViktorHofer
Copy link
Member Author

We already consume vstest via dependency flow which means getting live updates which aren't bound to VS releases. Additionally to that, we are actively collaborating with the vstest team (cc @nohwnd) and benefiting from each other.

@ViktorHofer ViktorHofer requested a review from safern March 23, 2020 15:55
@ViktorHofer ViktorHofer self-assigned this Mar 23, 2020
@@ -2,19 +2,15 @@
<PropertyGroup>
<!-- Microsoft.Net.Test.Sdk brings a lot of satellite assemblies in. -->
<SatelliteResourceLanguages>en</SatelliteResourceLanguages>
<TrxTestResultsName>testResults.trx</TrxTestResultsName>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know if the helix sdk supports .trx files to show the test results in AzDo, submitting to kusto and failing the build? If not we should request a feature for when we want to move to use this as our default test runner. cc: @alexperovich

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Helix supports trx files for uploading to azure devops. Kusto is only supported for xunit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a lot of people use Kusto for metrics and what not as the AzDo APIs aren't that great to get test results and search through them. Would it be possible to add Kusto from a trx file?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The support for processing xunit results into kusto is on the server side for helix and extending that for other formats would be a very large an potentially disruptive change. The azure devops uploading code is in arcade and very easy to change so it supports many formats. My understanding is that we want to consolidate everything to azure devops and make that experience better, rather than continue investing in our custom reporting. @Chrisboh may have more context.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I knew that trx is supported but I didn't know about Kusto only working with the xunit xml format.

@@ -1,10 +1,13 @@
<Project>
<PropertyGroup>
<RunCommand>"$(RunScriptHost)"</RunCommand>
<RunCommand>"$(DotNetTool)"</RunCommand>
<RunCommand Condition="'$(ContinuousIntegrationBuild)' == 'true'">dotnet</RunCommand>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to understand, could you explain what's the difference in between DotNetTool and dotnet and why does it has to be different in CI?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DotNetTool is a property set by Arcade, mean it's available during the test invocation on your machine. On a different machine you don't want to hardcode the path into the RunTemplate.cmd/sh. There is more work required here anyway to make this work on CI.

My goal in a subsequent PR is to just use dotnet here instead and then for local local development, modify PATH to point to the RunScriptHost if not on CI. As said, I don't want to do this now as this just enables VSTest optionally and not on CI.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another blocker for enabling VSTest in CI is #1980.

@@ -0,0 +1,5 @@
{
"diagnosticMessages": false,
"longRunningTestSeconds": 120,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this support is now built-in in VSTest?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah but I still need to play around with it. There's a blame mode which creates a sequence.xml which tells you which tests have run and which haven't but I don't know if that mode slows down the test invocation. cc @nohwnd

@@ -118,7 +118,6 @@
<XUnitVersion>2.4.1</XUnitVersion>
<TraceEventVersion>2.0.5</TraceEventVersion>
<NewtonsoftJsonVersion>12.0.3</NewtonsoftJsonVersion>
<XUnitXmlTestLoggerVersion>2.1.26</XUnitXmlTestLoggerVersion>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this removed? Just left-over?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this was introduced only for vstest in a past PR but actually we don't need it.

Copy link
Member

@safern safern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for solving all my questions.

@ViktorHofer
Copy link
Member Author

Issue was #32377.

@ViktorHofer ViktorHofer merged commit 4a55b46 into master Mar 26, 2020
@ViktorHofer ViktorHofer deleted the DefaultVSTest branch March 26, 2020 13:36
@ghost ghost locked as resolved and limited conversation to collaborators Dec 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants