Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specs: add global timeout to individual specs on mt #9865

Merged
merged 1 commit into from
Nov 17, 2020

Conversation

bcardiff
Copy link
Member

Now and then the preview_mt specs fails.
Initially they used to run in verbose mode to get a notion at which point that happens.

I figure we can add some timeout detection to detect specs that will hang due to events that will not happen. Tight loops will not be aborted, but those should be present in single-thread also.

@straight-shoota
Copy link
Member

If the spec still finishes at some point (after SPEC_TIMEOUT), it would report twice, wouldn't it? Not sure that's really any problem, just want to bring it up.

@bcardiff
Copy link
Member Author

bcardiff commented Nov 4, 2020

It's true that some examples could be reported twice. They will as long as the main fiber didn't exit. The following example has one of each.

require "spec"
require "./spec/support/mt_abort_timeout"

it "hangs (1)" do
  sleep 20
  raise "oh no"
end

it "hangs (2)" do
  sleep 20
end
$ bin/crystal spec -Dpreview_mt hang-spec.ign.cr 
FEF

Failures:

  1) hangs (1)
     Failure/Error: it "hangs (1)" do

       spec timeout

     # hang-spec.ign.cr:4

  2) hangs (2)
     Failure/Error: it "hangs (2)" do

       spec timeout

     # hang-spec.ign.cr:9

  3) hangs (1)

       oh no (Exception)
         from hang-spec.ign.cr:6:3 in '->'
         from src/primitives.cr:255:3 in 'internal_run'
         from src/spec/example.cr:32:73 in '->'
         from src/primitives.cr:255:3 in 'run'
         from spec/support/mt_abort_timeout.cr:10:7 in '->'
         from src/primitives.cr:255:3 in 'run'
         from src/fiber.cr:92:34 in '->'
       

Finished in 30.01 seconds
3 examples, 2 failures, 1 errors, 0 pending

Failed examples:

crystal spec hang-spec.ign.cr:4 # hangs (1)
crystal spec hang-spec.ign.cr:9 # hangs (2)
crystal spec hang-spec.ign.cr:4 # hangs (1)

But I'm fine with that.

The alternative would be to add some state Spec::Example to know if it has "ended" and hence skip later reports. But on the other hand, if the spec finishes (successfully or with error) it might be useful to know if the timeout is too low.

@straight-shoota
Copy link
Member

I suppose the double reporting is fine. A downside however is that the tally is incorrect. It reports 3 examples, although there are only 2 (the one that errors was counted twice).

@bcardiff
Copy link
Member Author

bcardiff commented Nov 4, 2020

Yes, I see that as part of the reporting.

The other alternative is to go back to verbose mode on preview_mt spec.

But it will offer less information IMO. With this PR we allow the rest of the specs to run and avoid hanging them because 1 got stuck. Plus, we will know which one hanged in a nice format.

@straight-shoota straight-shoota added this to the 1.0.0 milestone Nov 15, 2020
@bcardiff bcardiff merged commit dca3cda into crystal-lang:master Nov 17, 2020
@bcardiff
Copy link
Member Author

Note: The "HTTP::Server #bind fails after close" spec at spec/std/http/server/server_spec.cr:200 timeout recently https://app.circleci.com/pipelines/github/crystal-lang/crystal/4678/workflows/a9809af7-db21-4c67-96e9-02628f00b6ec/jobs/55559/steps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants