Regular --where test filter with quotes or spaces cannot be passed via @ FILE option

[k15tfu]

Hi!

It turned out that == operator for test names with quotes or spaces cannot be passed via @FILE, unlike the case when it is passed as command line arguments:

using NUnit.Framework;

namespace nunit_testfilter_csharp
{
    [TestFixture]
    public class Class1
    {
        [TestCase("with spaces")]
        public void Test(string name)
        {
            Assert.AreEqual(name, "with spaces");
        }
    }
}

and run with:

>TYPE ...\NunitWhereArg.txt
"test==/nunit_testfilter_csharp.Class1.Test(\"with spaces\")/"
>tools\nunit3-console.exe ...\bin\Debug\nunit_testfilter_csharp.dll --where @...\NunitWhereArg.txt  --noresult
NUnit Console 3.16.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Tuesday, August 15, 2023 1:09:40 PM

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    C:\p\test\nunit_testfilter_csharp\bin\Debug\nunit_testfilter_csharp.dll
    with
    spaces\")/"

System.ArgumentException : Illegal characters in path.

--ArgumentException
Illegal characters in path.
   at System.IO.LongPathHelper.Normalize(String path, UInt32 maxPathLength, Boolean checkInvalidCharacters, Boolean expandShortPaths)
   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
   at System.IO.Path.GetFullPathInternal(String path)
   at NUnit.Engine.TestPackage..ctor(String filePath)
   at NUnit.Engine.TestPackage..ctor(IList`1 testFiles)
   at NUnit.ConsoleRunner.ConsoleRunner.MakeTestPackage(ConsoleOptions options)
   at NUnit.ConsoleRunner.ConsoleRunner.Execute()
   at NUnit.ConsoleRunner.Program.Main(String[] args)

but it works when passing --where argument as parameters of a command line:

>...\tools\nunit3-console.exe ...\bin\Debug\nunit_testfilter_csharp.dll --where "test==/nunit_testfilter_csharp.Class1.Test(\"with spaces\")/" --noresult
NUnit Console 3.16.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Tuesday, August 15, 2023 1:15:44 PM

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    C:\p\test\nunit_testfilter_csharp\bin\Debug\nunit_testfilter_csharp.dll

Test Filters
    Where: test==/nunit_testfilter_csharp.Class1.Test("with spaces")/


Run Settings
    DisposeRunners: True
    WorkDirectory: ...
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.7.2
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    TargetRuntimeFramework: net-4.7.2
    NumberOfTestWorkers: 20

Test Run Summary
  Overall result: Passed
  Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2023-08-15 11:15:44Z
    End time: 2023-08-15 11:15:44Z
    Duration: 0.552 seconds

NUnit Console 3.16.3
NUnit 3.13.3

Linked issues: #94, #1181.

[CharliePoole]

From the documentation for @file...

"Specifies the name (or path) of a FILE containing additional command-line arguments to be interpolated at the point where the @file expression appears. Each line in the file represents a separate command-line argument."

In this case, your aren't passing the full argument but only a part of it...

Instead try either of the following in your file...

--where test==/nunit_testfilter_csharp.Class1.Test("with spaces")/
--test /nunit_testfilter_csharp.Class1.Test("with spaces")/

The second line uses the --test option rather than the --where option.

Note that double quotes need to be escaped with a backslash on the command-line but not in the file. That's a feature. :-)

[k15tfu]

@CharliePoole Thank you for the quick response!

Instead try either of the following in your file...

The first line doesn't work either:

>TYPE ...\NunitWhereArg.txt
--where test==/nunit_testfilter_csharp.Class1.Test("with spaces")/
>...\tools\nunit3-console.exe ...\bin\Debug\nunit_testfilter_csharp.dll @...\NunitWhereArg.txt --noresult
NUnit Console 3.16.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Tuesday, August 15, 2023 6:54:50 PM

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    C:\p\test\nunit_testfilter_csharp\bin\Debug\nunit_testfilter_csharp.dll
    spaces")/

System.ArgumentException : Illegal characters in path.

--ArgumentException
Illegal characters in path.
   at System.IO.LongPathHelper.Normalize(String path, UInt32 maxPathLength, Boolean checkInvalidCharacters, Boolean expandShortPaths)
   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
   at System.IO.Path.GetFullPathInternal(String path)
   at NUnit.Engine.TestPackage..ctor(String filePath)
   at NUnit.Engine.TestPackage..ctor(IList`1 testFiles)
   at NUnit.ConsoleRunner.ConsoleRunner.MakeTestPackage(ConsoleOptions options)
   at NUnit.ConsoleRunner.ConsoleRunner.Execute()
   at NUnit.ConsoleRunner.Program.Main(String[] args)

Each line in the file represents a separate command-line argument

This was my second question, if we are talking about command line parameters:

...\tools\nunit3-console.exe ...\bin\Debug\nunit_testfilter_csharp.dll --where "test==/nunit_testfilter_csharp.Class1.Test(\"with spaces\")/" --noresult

, I'd say --where and test==/nunit_testfilter_csharp.Class1.Test("with spaces")/ are two separate arguments (not one), so in this case I should be able to pass only the latter argument via @FILE but it also doesn't work.

[CharliePoole]

I no longer work on the NUnit project, but it does look as if the particular expression --where test==... may have a bug, which the @nunit/framework-team could address. I do know that --test... has special code to handle double quotes used by NUnit in its test names.

WRT what is meant by an argument, I must admit that the docs could have been more specific. What each line represents is actuall a separate argument as handled internally by NUnit. Depending on how you write it, this may be a single command-line argument or two arguments. For example, compare --where cat=MyCategory and --where:cat=MyCategory.

Additionally, this handling will also vary between operating systems. Consider, for example that most Linux systems handle single quotes specially on the command-line while Windows doesn't. One of the ideas behind both @FILE and --test-list=FILE) was to provide a way of coding arguments that would work across different operating systems. As a result, the syntax differs from what is acceptable in any particular operating system.

[CharliePoole]

Switching this to be purely a documentation issue. It would be a bug if we really wanted to be able to put arbitrary text in the file and have it combine with what was on the command-line, but that has never been the goal of this feature. OTOH, we need to be much clearer about how it works in the docs.

[CharliePoole]

This issue has been resolved in version 3.18.2

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org