-
Notifications
You must be signed in to change notification settings - Fork 320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Traits (Category) should differentiate test cases even when they have same data name #5097
Comments
Looking at your code I don't see anything vstest specific, this looks like a problem specific to NUnit. Have you tried filing the issue in Nunit? https://github.com/nunit/nunit |
Well I have created an issue in nunit3-vs-adapter, here: And they told me, that it is not on their side, but it is how Test Explorer is identifying tests by FQN. But couldn't category be put into that FQN? So just by having different category (traits) on a testcasesource, it could have the very same name and would be properly found as different test cases? Don't know why Category is not put into that FQN to differentiate tests, it doesn't seems right to me, and it makes working with testCaseSources harder! |
This would be a big change, but you can file an issue to TestExplorer by pressing the feedback button in VS. It would also require nunit to add that information to the fully qualified name so it becames distinct. |
Ok thanks, I have filed an issue there :) |
This work could also try to solve another problem, that is currently there. We are discussing it with @OsirisTerje at this issue: As I said, it could only be very logical to even add another attributes besides category, as follows:
or even more attributes, if someone would need them. All of these would be part of FQN, to differentiate tests from different datasources, even when they have same dataname. And you could of course group by all of these 4 attributes independently in Test Explorer !!! |
Hi @nohwnd, so what do you think about this? PS: If you agree with me, that this change won't do any harm to existing behavior, and everything would work exactly the same after this change, would you please upvote this feature request? |
I doubt it, every time we touch the fqn or the code that generates IDs we need to make sure that all consumers know about the change and accomodate it. |
But what behavior would be changed? Other than repairing a "bug" I was talking about? Just to be sure, do you understand, why I am talking about it as a bug? I am not so sure, if you really understand what I am talking about. For me, there is absolutely no reason why I should write to the name of the test, the same thing I wrote to Category, just to make the TestDataSource to appear in Test Explorer. Once again, this is not correct at all, and I am really surprised, that it is still working like this. Did you carefully looked at my example at the very beginning? Because I doubt it. If you look at it carefully, you will understand, that this is an issue with an EASY FIX, that doesn't break anything... THERE IS NOBODY, THAT NEEDS TO ACCOMODATE TO THIS CHANGE !!! PS: Prosím ťa, naozaj sa nad tým trochu zamysli, či by sa to naozaj nedalo úplne v kľude spraviť, pre mňa je to veľmi dôležité. |
If we add category to the FQN that is parsed by VS Test Explorer, then they will need to change their regex that parses it and accommodate the difference. If we do that we should also add it to the ID generator, which would change the IDs generated for the test cases, and that would break TestCase functionality in Azure Devops, the ID for test cases that have category would now be different than before without any user change. There is a bunch of metadata available already to test explorer (like traits), but category is not part of the hierarchy that we have, that is historically a container, namespace, class, method, data. And we cannot easily change this. |
"There is a bunch of metadata available already to test explorer (like traits), but category is not part of the hierarchy that we have" Alright, now I understand you well. But still have few more questions:
So I still have this vision, of having TraitGroups, Traits and Subtraits and then all 3 of these could be configurable from .runsettings to be part of FQN, if someone need it, like me :D So what do you think? I don't want to discuss this forever and take your precious time, but this was like my very last question, if you say NO to both of these questions, I will give up :D |
As said here and in nunit repo we won't be changing the fqn to include category name. |
Hi there,
I have found, that only because same dataset has the very same name as another dataset coming from different datasource, it is not discovered in VS Test Explorer. But it should, because by putting different Category (from NUnit), I already want to clearly say, that that datasource is actually different.
Even when I use Traits to group my tests, these tests are never discovered when having the same name.
I have made simple NUnit Test project with this simple file, which should absolutely clearly illustrate what I mean:
https://gist.github.com/PeterHevesi/aa1392c27000985c13c35dff9653f83a
The text was updated successfully, but these errors were encountered: