Problems running tests on macOS

[adetaylor]

Per discussion in #1975 and #1993, tests aren't working for me.

cargo test --all works in the root directory, but as soon as I do cc tests/expectations && cargo test I get failures. This is what the PR builder bots do, and I want to replicate their results.

Here's what I get:

$ git clone git@github.com:rust-lang/rust-bindgen.git
Cloning into 'rust-bindgen'...
remote: Enumerating objects: 75, done.
remote: Counting objects: 100% (75/75), done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 24584 (delta 34), reused 18 (delta 6), pack-reused 24509
Receiving objects: 100% (24584/24584), 13.77 MiB | 14.60 MiB/s, done.
Resolving deltas: 100% (18702/18702), done.
$ cd rust-bindgen
$ git log -n 1
commit b1c417882662c26398aae6c961b8a503ea8a27e7 (HEAD -> master, origin/master, origin/HEAD)
Author: Saleem Rashid <dev@saleemrashid.com>
Date:   Sun Feb 7 22:27:45 2021 +0000

    Run tests on AArch64 using Debian multiarch and QEMU
adetaylor-macbookpro:rust-bindgen adetaylor$ rustup default stable
info: using existing install for 'stable-x86_64-apple-darwin'
info: default toolchain set to 'stable-x86_64-apple-darwin'

  stable-x86_64-apple-darwin unchanged - rustc 1.50.0 (cb75ad5db 2021-02-10)

$ cd tests/expectations && cargo test
    Updating crates.io index
   Compiling libc v0.2.86
   Compiling tests_expectations v0.1.0 (/Users/adetaylor/dev/rust-bindgen/tests/expectations)
   Compiling cfg-if v1.0.0
   Compiling block v0.1.6
   Compiling libloading v0.6.7
   Compiling malloc_buf v0.0.6
   Compiling objc v0.2.7
warning: `extern` block uses type `Block<(i32, i32), i32>`, which is not FFI-safe
  --> tests/block_return_type.rs:11:22
   |
11 |     pub fn func() -> _bindgen_ty_id_4;
   |                      ^^^^^^^^^^^^^^^^ not FFI-safe
   |
   = note: `#[warn(improper_ctypes)]` on by default
   = note: composed only of `PhantomData`

warning: 1 warning emitted

warning: structure field `Cch` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:27:9
   |
27 |     pub Cch: Char,
   |         ^^^ help: convert the identifier to snake case (notice the capitalization): `cch`
   |
   = note: `#[warn(non_snake_case)]` on by default

warning: structure field `Cu` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:28:9
   |
28 |     pub Cu: UChar,
   |         ^^ help: convert the identifier to snake case (notice the capitalization): `cu`

warning: structure field `Cd` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:29:9
   |
29 |     pub Cd: SChar,
   |         ^^ help: convert the identifier to snake case (notice the capitalization): `cd`

warning: structure field `Ccch` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:30:9
   |
30 |     pub Ccch: Char,
   |         ^^^^ help: convert the identifier to snake case (notice the capitalization): `ccch`

warning: structure field `Ccu` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:31:9
   |
31 |     pub Ccu: UChar,
   |         ^^^ help: convert the identifier to snake case (notice the capitalization): `ccu`

warning: structure field `Ccd` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:32:9
   |
32 |     pub Ccd: SChar,
   |         ^^^ help: convert the identifier to snake case (notice the capitalization): `ccd`

warning: function `bindgen_test_layout_Test` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:35:4
   |
35 | fn bindgen_test_layout_Test() {
   |    ^^^^^^^^^^^^^^^^^^^^^^^^ help: convert the identifier to snake case: `bindgen_test_layout_test`

warning: 7 warnings emitted

warning: unused `#[macro_use]` import
 --> tests/objc_sel_and_id.rs:9:1
  |
9 | #[macro_use]
  | ^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

error[E0204]: the trait `Copy` may not be implemented for this type
  --> tests/objc_interface_type.rs:31:17
   |
31 | #[derive(Debug, Copy, Clone)]
   |                 ^^^^
32 | pub struct FooStruct {
33 |     pub foo: Foo,
   |     ------------ this field does not implement `Copy`
   |
   = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

For more information about this error, try `rustc --explain E0204`.
error: could not compile `tests_expectations`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
warning: 1 warning emitted

error: build failed

Problems also occur if I use the version from #1993

$ git fetch origin test-version-modern
From github.com:rust-lang/rust-bindgen
 * branch              test-version-modern -> FETCH_HEAD
hecadetaylor-macbookpro:expectations adetaylor$ git checkout FETCH_HEAD
Note: switching to 'FETCH_HEAD'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3ffb021e tests: Guess modern libclang version when we fail to parse a version.
$ cargo test
   Compiling tests_expectations v0.1.0 (/Users/adetaylor/dev/rust-bindgen/tests/expectations)
warning: structure field `Cch` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:27:9
   |
27 |     pub Cch: Char,
   |         ^^^ help: convert the identifier to snake case (notice the capitalization): `cch`
   |
   = note: `#[warn(non_snake_case)]` on by default

warning: structure field `Cu` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:28:9
   |
28 |     pub Cu: UChar,
   |         ^^ help: convert the identifier to snake case (notice the capitalization): `cu`

warning: structure field `Cd` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:29:9
   |
29 |     pub Cd: SChar,
   |         ^^ help: convert the identifier to snake case (notice the capitalization): `cd`

warning: structure field `Ccch` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:30:9
   |
30 |     pub Ccch: Char,
   |         ^^^^ help: convert the identifier to snake case (notice the capitalization): `ccch`

warning: structure field `Ccu` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:31:9
   |
31 |     pub Ccu: UChar,
   |         ^^^ help: convert the identifier to snake case (notice the capitalization): `ccu`

warning: structure field `Ccd` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:32:9
   |
32 |     pub Ccd: SChar,
   |         ^^^ help: convert the identifier to snake case (notice the capitalization): `ccd`

warning: function `bindgen_test_layout_Test` should have a snake case name
  --> tests/test_mixed_header_and_header_contents.rs:35:4
   |
35 | fn bindgen_test_layout_Test() {
   |    ^^^^^^^^^^^^^^^^^^^^^^^^ help: convert the identifier to snake case: `bindgen_test_layout_test`

warning: 7 warnings emitted

warning: use of deprecated function `std::mem::uninitialized`: use `mem::MaybeUninit` instead
  --> tests/constructors_1_33.rs:42:33
   |
42 |         let mut __bindgen_tmp = ::std::mem::uninitialized();
   |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(deprecated)]` on by default

warning: use of deprecated function `std::mem::uninitialized`: use `mem::MaybeUninit` instead
  --> tests/constructors_1_33.rs:48:33
   |
48 |         let mut __bindgen_tmp = ::std::mem::uninitialized();
   |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^

warning: use of deprecated function `std::mem::uninitialized`: use `mem::MaybeUninit` instead
  --> tests/constructors_1_33.rs:78:33
   |
78 |         let mut __bindgen_tmp = ::std::mem::uninitialized();
   |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^

warning: 3 warnings emitted

error[E0412]: cannot find type `WhitelistMe` in this scope
  --> tests/objc_whitelist.rs:13:42
   |
13 | impl WhitelistMe_InterestingCategory for WhitelistMe {}
   |                                          ^^^^^^^^^^^ not found in this scope

warning: unused `#[macro_use]` import
 --> tests/objc_whitelist.rs:9:1
  |
9 | #[macro_use]
  | ^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

error: aborting due to previous error; 1 warning emitted

For more information about this error, try `rustc --explain E0412`.
error: could not compile `tests_expectations`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed

clang diagnostics, in case they're relevant:

$ clang --version
clang version 10.0.1 
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm/bin
$ llvm-config --version
10.0.1
$ which llvm-config
/usr/local/opt/llvm/bin/llvm-config

which presumably is the version from homebrew, and perhaps that's the problem.

[adetaylor]

However I have good news! On my Linux box, cd test/expectations && cargo test now passes. I suspect that previous failures on my Linux box were conceivably because I had an unfortunate version of nightly Rust, or maybe I had some old artifacts around which git clean -ffdx got rid of. In any case, I can now replicate the test failures received by the bots on #1975 and should be able to make (slow) progress.

[emilio]

Ahá, so those are the objective-c tests... I guess that's #1954 then.

TLDR: Historically we didn't run cd test/expectations && cargo test on Mac, because of an oversight. So a couple objective-c changes landed which broke the tests, and I noticed when moving CI out of Travis. I couldn't investigate more because I didn't have a mac around, but we should probably disable those tests in the interim, because as you said otherwise replicating the test runs in the bots is not really possible.

[emilio]

Does this (after updating the tests) help?

diff --git a/tests/tests.rs b/tests/tests.rs
index 7bf2c65c..e92d44ce 100644
--- a/tests/tests.rs
+++ b/tests/tests.rs
@@ -305,7 +305,8 @@ fn create_bindgen_builder(header: &PathBuf) -> Result<Builder, Error> {
                 .unwrap();
             flags.extend(extra_flags.into_iter());
         } else if line.contains("bindgen-osx-only") {
-            let prepend_flags = ["--raw-line", "#![cfg(target_os=\"macos\")]"];
+            // FIXME(#1954): Objective-C tests are broken.
+            let prepend_flags = ["--raw-line", "#![cfg(target_os=\"nonexistent\")]"];
             flags = prepend_flags
                 .iter()
                 .map(ToString::to_string)

Obviously not ideal though...

[emilio]

cargo test --all works in the root directory, but as soon as I do cc tests/expectations && cargo test I get failures. This is what the PR builder bots do, and I want to replicate their results.

This is with or without #1993? In #1991 the failures happened on cargo test.

[adetaylor]

Does this (after updating the tests) help?

It doesn't seem to help. I still get the same cannot find type WhitelistMe in this scope errors.

This is with or without #1993?

Testing now.

[adetaylor]

No matter whether or not I have the #1993 change, cargo test --all passes in the root but cd tests/expectations && cargo test fails.

I'm going to be away for a few days now so I'm afraid I won't be able to help with further reproduction/diagnosis for a while. But in any case, I'm unblocked because I can run my tests on Linux. Thanks for digging into it!

[adetaylor]

OK, I got a chance to dig into this again. The only problem I can still see remaining is #1954, so I'm going to close this. Thanks for help diagnosing!