Allow benchmark classes and methods to be executed using TestDriven.Net

[jcansdale]

This branch adds support for executing benchmarks using TestDriven.Net. To try it, simply reference the BenchmarkDotNet.TestDriven package and add an attribute like this:

[assembly: BenchmarkTestRunner(typeof(FastAndDirtyConfig))]

If you have TestDriven.Net installed, you can then right-click Run Test(s) inside any class or method that contains a [Benchmark] attribute.

I've added one of these attributes to the BenchmarkDotNet.Samples project so you can try it there:
https://github.com/jcansdale/BenchmarkDotNet/blob/master/samples/BenchmarkDotNet.Samples/Properties/AssemblyInfo.cs

Thanks for a great tool and let me know if you have any questions. 😄

[jcansdale]

I'm not sure why this build is broken:

  Project BenchmarkDotNet.TestDriven is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Project BenchmarkDotNet.TestDriven supports:
      - net45 (.NETFramework,Version=v4.5)
      - netstandard1.6 (.NETStandard,Version=v1.6)

I thought netcoreapp1.0 was pretty much a super-set of netstandard1.6. 😕

Any ideas?

[AndreyAkinshin]

Hi @jcansdale,

• I don't think that the TestDriven support should be placed in the core BenchmarkDotNet repo. If you want, you can create own repo and own NuGet package (like TestDriven.BenchmarkDotNet). You don't need BenchmarkDotNet source code for that, just add a dependency to the core package.
• Using FastAndDirtyConfig everywhere kills the main advantage of BenchmarkDotNet — good precision level, it's definitely a bad practice.
• Your code doesn't follow our coding conventions (you can look at the other projects in the solution): we don't start method names with a lowercase letter, we use xUnit instead of NUnit, and so on.

[mattwarren]

@AndreyAkinshin I understand not wanting this in the core repo, but what about hosting it under PerfDotNet?

Maybe a "Contrib" or "3rd party Integration" section or something like that? It would be nice to have things like this hosted in one place, so they're easy to find, what do you think?

[AndreyAkinshin]

@mattwarren, let me think about it.

[jcansdale]

I'm fortunate that the xUnit guys use TestDriven.Net, so dogfood an ITestRunner implementation as part of their core repo:
https://github.com/xunit/xunit/tree/master/src/xunit.runner.tdnet

For NUnit I maintain a repo as part of the NUnit organization:
https://github.com/nunit/nunit3-tdnet-adapter

This is what @mattwarren is suggesting and also works well.

As much as anything I want to give the BenchmarkDotNet team an opportunity to beat my up if I'm doing it wrong or abusing your interfaces. 😉

Using FastAndDirtyConfig everywhere kills the main advantage of BenchmarkDotNet — good precision level, it's definitely a bad practice.

I certainly don't want to encourage bad practice, but have found the ability to quickly preview a benchmark to be useful. The alternative is to require the user to come up with their own quick benchmark attributes and remember to swap them out when they're done developing the benchmark.

[AndreyAkinshin]

Ok, it makes sense. Probably, a separated repo under PerfDotNet is a good idea.
Let me finish JobMutators (see #257 (comment)), it should allow to create a nice UI with abilities to run benchmarks in the dry and full modes (without FastAndDirty on the assembly level).