Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In/Out_channel Lin test revision #370

Merged
merged 4 commits into from
Jul 7, 2023
Merged

In/Out_channel Lin test revision #370

merged 4 commits into from
Jul 7, 2023

Conversation

jmid
Copy link
Collaborator

@jmid jmid commented Jun 21, 2023

This PR makes a few adjustments to the In/Out_channel Lin tests

  • it adds bindings requiring bytes which is now available
  • it increases the Lin Out_channel test count, as this is sometimes causing a CI red light with a failure to trigger
  • it renames the lin_tests_dsl_common_io library to lin_tests_spec_io to avoid a library/module-name mismatch (minor nit, admittedly)

To understand the Out_channel failure to trigger, I ran some local (Linux) stats.
Interestingly, an In_channel parallel issue is much easier to trigger than an Out_channel one:

  In_channel Domain 101 / 10000       
  Out_channel Domain 0 / 10000

These don't use Util.repeat though. Doing so with Util.repeat 50 (the default) brings the former up to 584 / 10000.
The latter is still crunching away after a couple of hours... 😬

Overall, this motivates the need for a higher test count and the occasional failures-to-trigger that we have observed.

@jmid
Copy link
Collaborator Author

jmid commented Jun 22, 2023

Ha! So as much as these stats are handy for determining slim error rates up front, they clearly lack in assessing times to shrink a found counterexample! (We could calculate stats for that too I suppose).

CI summary of b8cc4bf:

  • 3 runs timed out due to long Lin In_channel shrink runs:
    • 32-bit 5.1: 12260secs
    • Windows bytecode 5.1: 9105secs
    • macOS 5.1 9127.9secs
  • 3 time outs due to threadomain locks on Windows bytecode 5.1 and trunk 5.2 twice [ocaml5-issue] Windows failures on threadomain #203
  • 1 Cygwin 5.1 and trunk (5.2) part 1 time outs due to general slowness (Lin Atomic, Lin Bytes, ...)
  • 1 Cygwin trunk 5.2 part 2 time out on a 10764sec Lin ref int64 test with Thread shrink run Improve reproducibility of Lin/STM Thread modes #338
  • Linux s390x timed out with several tests taking 1000secs (due to load on the machine I suppose)

The former is news!
I strongly suspect this is caused by Lin fighting to shrink long strings underlying the bytes combinator.

@jmid
Copy link
Collaborator Author

jmid commented Jun 22, 2023

041c177 switches the bytes to use bytes_small akin to what we already do for string.

Locally, I'm observing runs of Lin Out_channel with an unusual execution profile:
Long runs up until finding a counterexample and high GC activity revealed by perf top (top-2):

  48.41%  lin_tests_dsl_domain.exe                       [.] pool_sweep
  14.37%  lin_tests_dsl_domain.exe                       [.] do_some_marking
  ...

I'm curious to see whether this behaviour shows up on the CI too.

@jmid
Copy link
Collaborator Author

jmid commented Jun 23, 2023

CI summary for 041c177:

The first one is related to this PR and indicated outstanding issues with the In_channel Lin test.

@jmid
Copy link
Collaborator Author

jmid commented Jul 6, 2023

Rebased on main with 5.1.0~beta1 CI tests.

@jmid
Copy link
Collaborator Author

jmid commented Jul 7, 2023

CI summary:

I'll merge as this PR

  • increases reproducibility of the Out_channel test
  • increases the covered surface by including bytes functions

We can then address excessive Lin DSL In_channel test with Domain shrinking runs separately.

@jmid jmid merged commit ec598c6 into main Jul 7, 2023
43 of 47 checks passed
@jmid jmid deleted the io-lin-revision branch July 7, 2023 16:42
@jmid jmid linked an issue Jul 10, 2023 that may be closed by this pull request
@jmid jmid mentioned this pull request Jan 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reliability of parallel Lin Out_channel tests
1 participant