Build errors after installing BenchMarkDotNet nightly build

[bradykelly]

Based on your recommendation in #1872, I have just installed the nightly build v0.10.3.13-nightly, and I now get the following build errors where I am trying to use BDN attributes GlobalSetup and GlobalCleanup:

• CS0246 The type or namespace name 'GlobalSetupAttribute' could not be found
• CS0246 The type or namespace name 'GlobalSetup' could not be found
• CS0246 The type or namespace name 'GlobalCleanupAttribute' could not be found
• CS0246 The type or namespace name 'GlobalCleanup' could not be found

I may have installed the incorrect nightly build version, but after adding the bdn-nightly package source, browsing that source with VS Nuget Package Manager didn't show any packages at all. So I used the package name and version given on your guide to installing nightly builds and installed the package via Package Manager Console.

[adamsitnik]

Hi @bradykelly

v0.10.3.13-nightly is a very old package, please try 0.13.1.1667

[bradykelly]

Thanks, things are looking better, but now it hangs on WorkloadJitting:

// Found 2 benchmarks:
//   Day12Runner.Part1: Job-MVONIF(IterationCount=10, LaunchCount=1, WarmupCount=5)
//   Day12Runner.Part2: Job-MVONIF(IterationCount=10, LaunchCount=1, WarmupCount=5)

// **************************
// Benchmark: Day12Runner.Part1: Job-MVONIF(IterationCount=10, LaunchCount=1, WarmupCount=5)
// *** Execute ***
// Launch: 1 / 1
// Execute: dotnet b88aa9da-8f03-425f-818d-d82d9b0a720c.dll --benchmarkName AdventOfCodeConsole.Runners._2021.Day12Runner.Part1 --job "IterationCount=10, LaunchCount=1, WarmupCount=5" --benchmarkId 0 in D:\Personal\Dotnet Projects\AdventOfCode\AdventOfCodeConsole\bin\Release\net6.0\b88aa9da-8f03-425f-818d-d82d9b0a720c\bin\Release\net6.0
// BeforeAnythingElse

// Benchmark Process Environment Information:
// Runtime=.NET 6.0.1 (6.0.121.56705), X64 RyuJIT
// GC=Concurrent Workstation
// Job: Job-EESWHB(IterationCount=10, LaunchCount=1, WarmupCount=5)

OverheadJitting  1: 1 op, 258600.00 ns, 258.6000 us/op
WorkloadJitting  1: 1 op, 34321600.00 ns, 34.3216 ms/op

It started, built its project fine, then started on the 1st benchmark and just hung for several minutes until I hit Ctrl+C. The code being benchmarked executes in about 1 second or less in Debug mode. This is my setup, if it helps:

[MemoryDiagnoser]
[SimpleJob(launchCount: 1, warmupCount: 5, targetCount: 10)]
public class DayRunnerBase<T> where T: IDay, new() 

DayRunnerBase has one method with GlobalSetup, one with GlobalCleanup and two with Benchmark attributes. What more information can I give you, and is there anything I can do to help diagnose this?

[bradykelly]

It seems it isn't hanging, just taking horrifically long doing something. I started my benchmark run about 45 minutes ago and so far have this output:

OverheadJitting  1: 1 op, 348500.00 ns, 348.5000 us/op
WorkloadJitting  1: 1 op, 36214600.00 ns, 36.2146 ms/op

WorkloadPilot    1: 2 op, 1826749886000.00 ns, 913.3749 s/op

That's 1800+ seconds, or 30 minutes, for WorkloadPilot, for benchmarking a method that runs in less than 0.5 seconds

[adamsitnik]

@bradykelly have you tried attaching a debugger to the benchmark process? (not the host process, but to the benchmark process). Here is how you can do it: https://benchmarkdotnet.org/articles/guides/troubleshooting.html#launch-a-debugger-from-the-benchmark-process-using-debugger-api

[bradykelly]

As told, under Attach a debugger from IDE I've added this to my GlobalSetup method:

while (!System.Diagnostics.Debugger.IsAttached)
    Thread.Sleep(TimeSpan.FromMilliseconds(100));

Then launched the process from within VS using Start without debugging and the BDN "child" process seems to be waiting, as it hasn't started any benchmarking work yet, like it has done prior to my edit. Then I attached to the only process I could find with a --benchmarkname command line argument, which seems to be the correct process because now BDN has started benchmarking and hung again.

Now I don't know what to do next. In the IDE that is attached to the process, I hit Pause and I get the following message:
Your app has entered a break state, but no code is currently executing that is supported by the selected debug engine (e.g. only native runtime code is executing).

Would it help any if I told you that BDN still works fine on other classes, but the class I am trying to benchmark is the first one I have tried where the benchmarked method uses recursion? I am currently busy with my next class that uses recursion and I will also try and benchmark that and see what happens.