add temp CI job to test syspolicy impact

[abathur]

Starting in Catalina, macOS runs a syspolicyd "assessment" that hits the network for each binary/script executable. It does cache these results, but Nix tends to introduce many "new" executables per build. (You can read more about this at NixOS/nix#3789).

This PR adds a temporary, redundant macOS job with these assessments disabled. I'm hoping you can adopt it for a few weeks to help me collect more data on how this affects real projects.

[junjihashimoto]

@abathur Thx!
I think we can try this pr in builds other than nix.

[abathur]

@junjihashimoto Not certain how to interpret this :)

Are you saying you are not willing to try this with the nix build?

Are you asking me to also apply this to stack-macos.yml as well?

My interest is specifically to evaluate how it is impacting Nix builds, but I don't mind a quick edit to add one for the stack-macos job.

[junjihashimoto]

@abathur Sorry for late reply.
I think that not only nix-build but general macos builds will be more efficient. Is it different?

[junjihashimoto]

I merge this pr for benchmark.

[abathur]

@abathur Sorry for late reply.

No worries.

I think that not only nix-build but general macos builds will be more efficient. Is it different?

I'm not sure how big the effect will be. I would guess so, but a few of the builds I'm collecting so far actually show the non-assessed builds as slower at the moment. I'm not really sure why. I'm surprised, but I'm trying to avoid looking at the data too much until samples build up a bit. :)

[junjihashimoto]

I have only seen two cases, but it seems that this does not improve the performance, but worsens it.

[abathur]

The data is pretty noisy, unfortunately.

It looks like I have 12 samples collected for hasktorch/hasktorch, and another 87 I collected (by enabling hourly tests on my fork for a few days to get more samples). Hasktorch has actually been one of the largest performance improvements in that sample, but some projects definitely appear to be unaffected or even slower.

I haven't quite puzzled out what's going on, but I've been picking at it when I have a little time.

My own project has been consistently, demonstrably faster on all 3 macbooks I have access to with the assessments disabled, but I've observed basically no difference with github actions. I had started to suspect that maybe Apple mitigated the impact of the issue in 10.15.5 (because I did all of the initial testing on 10.15.4, but all of the runs I've collected in CI have been after the 10.15.5 roll-out), but I took a little time this past week to prove that I was still seeing the difference locally.

I need to circle back to build some separate tests to sanity check the assessment behavior on CI.

If the job is causing you any delay or trouble, feel free to revert/remove it.