Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestPantsDaemonIntegration.test_pantsd_sigquit is flaky #7572

Closed
jsirois opened this issue Apr 15, 2019 · 0 comments · Fixed by #11223
Closed

TestPantsDaemonIntegration.test_pantsd_sigquit is flaky #7572

jsirois opened this issue Apr 15, 2019 · 0 comments · Fixed by #11223

Comments

@jsirois
Copy link
Contributor

jsirois commented Apr 15, 2019

Seen in nightly cron Ci job on 'Integration tests - shard 1 (Py3.7 PEX)' shard:

                   Invalidated 1 target.
06:17:32 18:06       [run]
                     Only executing 2 of 27 total tests in shard 1 of 20
                     .F                                         [100%]
                     ==================== FAILURES ====================
                     _ TestPantsDaemonIntegration.test_pantsd_sigquit _
                     
                     self = <pants_test.pantsd.test_pantsd_integration.TestPantsDaemonIntegration testMethod=test_pantsd_sigquit>
                     
                         def test_pantsd_sigquit(self):
                           self._assert_pantsd_keyboardinterrupt_signal(
                             signal.SIGQUIT,
                             regexps=[
                               '\\[INFO\\] Sending SIGQUIT to pantsd-runner with pid [0-9]+, waiting up to 5\\.0 seconds before sending SIGKILL\\.\\.\\.',
                               re.escape("\nSignal {signum} (SIGQUIT) was raised. Exiting with failure.\n"
                                         .format(signum=signal.SIGQUIT)),
                               """
                         Interrupted by user:
                         Interrupted by user over pailgun client!
                     >   $"""])
                     
                     .pants.d/pyprep/sources/6e626b2a932c13ade8fdfe460aa59cbf57fc572d/pants_test/pantsd/test_pantsd_integration.py:519: 
                     _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                     .pants.d/pyprep/sources/6e626b2a932c13ade8fdfe460aa59cbf57fc572d/pants_test/pantsd/test_pantsd_integration.py:486: in _assert_pantsd_keyboardinterrupt_signal
                         assertRegex(self, waiter_run.stderr_data, regexp)
                     .pants.d/pyprep/sources/6e626b2a932c13ade8fdfe460aa59cbf57fc572d/pants_test/testutils/py2_compat.py:17: in assertRegex
                         self.assertRegex(text, expected_regex, msg)
                     E   AssertionError: Regex didn't match: '\\\nSignal\\ 3\\ \\(SIGQUIT\\)\\ was\\ raised\\.\\ Exiting\\ with\\ failure\\.\\\n' not found in '06:18:10 [INFO] Sending SIGQUIT to pantsd-runner with pid 17788, waiting up to 5.0 seconds before sending SIGKILL...\nInterrupted by user:\nInterrupted by user over pailgun client!\n'
                     -------------- Captured stdout call --------------
                     
                     pantsd log is /home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pantsd/pantsd.log
                     >>> config: 
                     {'GLOBAL': {'enable_pantsd': True, 'watchman_socket_path': '/tmp/watchman.17219.sock', 'level': 'info', 'pants_subprocessdir': '/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.pids'}}
                     
                     
                     running: ./pants kill-pantsd (config={'GLOBAL': {'enable_pantsd': True, 'watchman_socket_path': '/tmp/watchman.17219.sock', 'level': 'info', 'pants_subprocessdir': '/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.pids'}}) (extra_env={})
                     
                     completed in 4.980223655700684 seconds
                     PantsDaemonMonitor: pid is 17765 is_alive=True
                     logs/exceptions.17689.log +++ 
                     logs/exceptions.17689.log >>> timestamp: 2019-04-15T06:18:06.137914
                     logs/exceptions.17689.log >>> process title: pantsd [/home/travis/build/pantsbuild/pants]
                     logs/exceptions.17689.log >>> sys.argv: ['/home/travis/build/pantsbuild/pants/pants.pex', '--no-pantsrc', '--pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d', '--kill-nailguns', '--print-exception-stacktrace=True', '--pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini', 'kill-pantsd']
                     logs/exceptions.17689.log >>> pid: 17689
                     logs/exceptions.17689.log >>> Signal 15 (SIGTERM) was raised. Exiting with failure.
                     logs/exceptions.17689.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 193, in _run_module_as_main
                     logs/exceptions.17689.log >>>     "__main__", mod_spec)
                     logs/exceptions.17689.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 85, in _run_code
                     logs/exceptions.17689.log >>>     exec(code, run_globals)
                     logs/exceptions.17689.log >>>   File "__main__.py", line 25, in <module>
                     logs/exceptions.17689.log >>>     bootstrap_pex(__entry_point__)
                     logs/exceptions.17689.log >>>   File ".bootstrap/_pex/pex_bootstrapper.py", line 170, in bootstrap_pex
                     logs/exceptions.17689.log >>>     pex.PEX(entry_point).execute()
                     logs/exceptions.17689.log >>>   File ".bootstrap/_pex/pex.py", line 349, in execute
                     logs/exceptions.17689.log >>>     exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
                     logs/exceptions.17689.log >>>   File ".bootstrap/_pex/pex.py", line 281, in _wrap_coverage
                     logs/exceptions.17689.log >>>     return runner(*args)
                     logs/exceptions.17689.log >>>   File ".bootstrap/_pex/pex.py", line 312, in _wrap_profiling
                     logs/exceptions.17689.log >>>     return runner(*args)
                     logs/exceptions.17689.log >>>   File ".bootstrap/_pex/pex.py", line 394, in _execute
                     logs/exceptions.17689.log >>>     return self.execute_entry(self._pex_info.entry_point)
                     logs/exceptions.17689.log >>>   File ".bootstrap/_pex/pex.py", line 506, in execute_entry
                     logs/exceptions.17689.log >>>     return runner(entry_point)
                     logs/exceptions.17689.log >>>   File ".bootstrap/_pex/pex.py", line 528, in execute_pkg_resources
                     logs/exceptions.17689.log >>>     return runner()
                     logs/exceptions.17689.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 87, in main
                     logs/exceptions.17689.log >>>     PantsLoader.run()
                     logs/exceptions.17689.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 83, in run
                     logs/exceptions.17689.log >>>     cls.load_and_execute(entrypoint)
                     logs/exceptions.17689.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 76, in load_and_execute
                     logs/exceptions.17689.log >>>     entrypoint_main()
                     logs/exceptions.17689.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 474, in launch
                     logs/exceptions.17689.log >>>     PantsDaemon.Factory.create(OptionsBootstrapper.create()).run_sync()
                     logs/exceptions.17689.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 407, in run_sync
                     logs/exceptions.17689.log >>>     self._run_services(self._services)
                     logs/exceptions.17689.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 362, in _run_services
                     logs/exceptions.17689.log >>>     service_thread.join(self.JOIN_TIMEOUT_SECONDS)
                     logs/exceptions.17689.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/threading.py", line 1036, in join
                     logs/exceptions.17689.log >>>     self._wait_for_tstate_lock(timeout=max(timeout, 0))
                     logs/exceptions.17689.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
                     logs/exceptions.17689.log >>>     elif lock.acquire(block, timeout):
                     logs/exceptions.17689.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/base/exception_sink.py", line 74, in handle_sigterm
                     logs/exceptions.17689.log >>>     raise self.SignalHandledNonLocalExit(signum, 'SIGTERM')
                     logs/exceptions.17689.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/base/exception_sink.py", line 67, in __init__
                     logs/exceptions.17689.log >>>     self.traceback_lines = traceback.format_stack()
                     logs/exceptions.17689.log >>> 
                     logs/exceptions.17689.log >>> 
                     logs/exceptions.17689.log >>> timestamp: 2019-04-15T06:18:06.139063
                     logs/exceptions.17689.log >>> process title: pantsd [/home/travis/build/pantsbuild/pants]
                     logs/exceptions.17689.log >>> sys.argv: ['/home/travis/build/pantsbuild/pants/pants.pex', '--no-pantsrc', '--pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d', '--kill-nailguns', '--print-exception-stacktrace=True', '--pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini', 'kill-pantsd']
                     logs/exceptions.17689.log >>> pid: 17689
                     logs/exceptions.17689.log >>> Exception caught: (builtins.ValueError)
                     logs/exceptions.17689.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 193, in _run_module_as_main
                     logs/exceptions.17689.log >>>     "__main__", mod_spec)
                     logs/exceptions.17689.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 85, in _run_code
                     logs/exceptions.17689.log >>>     exec(code, run_globals)
                     logs/exceptions.17689.log >>>   File "__main__.py", line 25, in <module>
                     logs/exceptions.17689.log >>>     bootstrap_pex(__entry_point__)
                     logs/exceptions.17689.log >>>   File ".bootstrap/_pex/pex_bootstrapper.py", line 170, in bootstrap_pex
                     logs/exceptions.17689.log >>>     pex.PEX(entry_point).execute()
                     logs/exceptions.17689.log >>>   File ".bootstrap/_pex/pex.py", line 368, in execute
                     logs/exceptions.17689.log >>>     sys.stderr.flush()
                     logs/exceptions.17689.log >>> 
                     logs/exceptions.17689.log >>> Exception message: I/O operation on closed file.
                     logs/exceptions.17689.log >>> 
                     logs/exceptions.17689.log --- 
                     logs/exceptions.17656.log +++ 
                     logs/exceptions.17656.log --- 
                     logs/exceptions.17732.log +++ 
                     logs/exceptions.17732.log --- 
                     logs/exceptions.17765.log +++ 
                     logs/exceptions.17765.log --- 
                     logs/exceptions.log +++ 
                     logs/exceptions.log >>> timestamp: 2019-04-15T06:18:06.137914
                     logs/exceptions.log >>> process title: pantsd [/home/travis/build/pantsbuild/pants]
                     logs/exceptions.log >>> sys.argv: ['/home/travis/build/pantsbuild/pants/pants.pex', '--no-pantsrc', '--pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d', '--kill-nailguns', '--print-exception-stacktrace=True', '--pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini', 'kill-pantsd']
                     logs/exceptions.log >>> pid: 17689
                     logs/exceptions.log >>> Signal 15 (SIGTERM) was raised. Exiting with failure.
                     logs/exceptions.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 193, in _run_module_as_main
                     logs/exceptions.log >>>     "__main__", mod_spec)
                     logs/exceptions.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 85, in _run_code
                     logs/exceptions.log >>>     exec(code, run_globals)
                     logs/exceptions.log >>>   File "__main__.py", line 25, in <module>
                     logs/exceptions.log >>>     bootstrap_pex(__entry_point__)
                     logs/exceptions.log >>>   File ".bootstrap/_pex/pex_bootstrapper.py", line 170, in bootstrap_pex
                     logs/exceptions.log >>>     pex.PEX(entry_point).execute()
                     logs/exceptions.log >>>   File ".bootstrap/_pex/pex.py", line 349, in execute
                     logs/exceptions.log >>>     exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
                     logs/exceptions.log >>>   File ".bootstrap/_pex/pex.py", line 281, in _wrap_coverage
                     logs/exceptions.log >>>     return runner(*args)
                     logs/exceptions.log >>>   File ".bootstrap/_pex/pex.py", line 312, in _wrap_profiling
                     logs/exceptions.log >>>     return runner(*args)
                     logs/exceptions.log >>>   File ".bootstrap/_pex/pex.py", line 394, in _execute
                     logs/exceptions.log >>>     return self.execute_entry(self._pex_info.entry_point)
                     logs/exceptions.log >>>   File ".bootstrap/_pex/pex.py", line 506, in execute_entry
                     logs/exceptions.log >>>     return runner(entry_point)
                     logs/exceptions.log >>>   File ".bootstrap/_pex/pex.py", line 528, in execute_pkg_resources
                     logs/exceptions.log >>>     return runner()
                     logs/exceptions.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 87, in main
                     logs/exceptions.log >>>     PantsLoader.run()
                     logs/exceptions.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 83, in run
                     logs/exceptions.log >>>     cls.load_and_execute(entrypoint)
                     logs/exceptions.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 76, in load_and_execute
                     logs/exceptions.log >>>     entrypoint_main()
                     logs/exceptions.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 474, in launch
                     logs/exceptions.log >>>     PantsDaemon.Factory.create(OptionsBootstrapper.create()).run_sync()
                     logs/exceptions.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 407, in run_sync
                     logs/exceptions.log >>>     self._run_services(self._services)
                     logs/exceptions.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 362, in _run_services
                     logs/exceptions.log >>>     service_thread.join(self.JOIN_TIMEOUT_SECONDS)
                     logs/exceptions.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/threading.py", line 1036, in join
                     logs/exceptions.log >>>     self._wait_for_tstate_lock(timeout=max(timeout, 0))
                     logs/exceptions.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
                     logs/exceptions.log >>>     elif lock.acquire(block, timeout):
                     logs/exceptions.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/base/exception_sink.py", line 74, in handle_sigterm
                     logs/exceptions.log >>>     raise self.SignalHandledNonLocalExit(signum, 'SIGTERM')
                     logs/exceptions.log >>>   File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/base/exception_sink.py", line 67, in __init__
                     logs/exceptions.log >>>     self.traceback_lines = traceback.format_stack()
                     logs/exceptions.log >>> 
                     logs/exceptions.log >>> 
                     logs/exceptions.log >>> timestamp: 2019-04-15T06:18:06.139063
                     logs/exceptions.log >>> process title: pantsd [/home/travis/build/pantsbuild/pants]
                     logs/exceptions.log >>> sys.argv: ['/home/travis/build/pantsbuild/pants/pants.pex', '--no-pantsrc', '--pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d', '--kill-nailguns', '--print-exception-stacktrace=True', '--pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini', 'kill-pantsd']
                     logs/exceptions.log >>> pid: 17689
                     logs/exceptions.log >>> Exception caught: (builtins.ValueError)
                     logs/exceptions.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 193, in _run_module_as_main
                     logs/exceptions.log >>>     "__main__", mod_spec)
                     logs/exceptions.log >>>   File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 85, in _run_code
                     logs/exceptions.log >>>     exec(code, run_globals)
                     logs/exceptions.log >>>   File "__main__.py", line 25, in <module>
                     logs/exceptions.log >>>     bootstrap_pex(__entry_point__)
                     logs/exceptions.log >>>   File ".bootstrap/_pex/pex_bootstrapper.py", line 170, in bootstrap_pex
                     logs/exceptions.log >>>     pex.PEX(entry_point).execute()
                     logs/exceptions.log >>>   File ".bootstrap/_pex/pex.py", line 368, in execute
                     logs/exceptions.log >>>     sys.stderr.flush()
                     logs/exceptions.log >>> 
                     logs/exceptions.log >>> Exception message: I/O operation on closed file.
                     logs/exceptions.log >>> 
                     logs/exceptions.log --- 
                     pantsd/pantsd.log +++ 
                     pantsd/pantsd.log >>> 06:18:05 [INFO] setting up service <pants.pantsd.service.fs_event_service.FSEventService object at 0x7f651c73ecf8>
                     pantsd/pantsd.log >>> 06:18:05 [INFO] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f651c73eef0>
                     pantsd/pantsd.log >>> 06:18:05 [INFO] watching invalidating files: set()
                     pantsd/pantsd.log >>> 06:18:05 [INFO] setting up service <pants.pantsd.service.pailgun_service.PailgunService object at 0x7f651c757d30>
                     pantsd/pantsd.log >>> 06:18:05 [INFO] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f651c757c88>
                     pantsd/pantsd.log >>> 06:18:05 [INFO] starting service <pants.pantsd.service.fs_event_service.FSEventService object at 0x7f651c73ecf8>
                     pantsd/pantsd.log >>> 06:18:05 [INFO] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f651c73eef0>
                     pantsd/pantsd.log >>> 06:18:05 [INFO] starting service <pants.pantsd.service.pailgun_service.PailgunService object at 0x7f651c757d30>
                     pantsd/pantsd.log >>> 06:18:05 [INFO] starting pailgun server on port 42517
                     pantsd/pantsd.log >>> 06:18:05 [INFO] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f651c757c88>
                     pantsd/pantsd.log >>> 06:18:05 [INFO] handling pailgun request: `./pants --no-pantsrc --pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d --kill-nailguns --print-exception-stacktrace=True --pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini kill-pantsd`
                     pantsd/pantsd.log >>> 06:18:05 [INFO] confirmed watchman subscription: {'clock': 'c:1555309083:17670:1:2', 'subscribe': 'all_files', 'version': '4.9.1'}
                     pantsd/pantsd.log >>> 06:18:05 [INFO] enqueuing 5066 changes for subscription all_files
                     pantsd/pantsd.log >>> 06:18:05 [INFO] confirmed watchman subscription: {'clock': 'c:1555309083:17670:1:4', 'subscribe': 'pantsd_pid', 'version': '4.9.1'}
                     pantsd/pantsd.log >>> 06:18:05 [INFO] enqueuing 1 changes for subscription pantsd_pid
                     pantsd/pantsd.log >>> 06:18:05 [INFO] pailgun request completed: `./pants --no-pantsrc --pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d --kill-nailguns --print-exception-stacktrace=True --pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini kill-pantsd`
                     pantsd/pantsd.log >>> 06:18:06 [WARN] Encountered error when trying to log this message: b'timestamp: 2019-04-15T06:18:06.138092\nSignal 15 (SIGTERM) was raised. Exiting with failure.\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 193, in _run_module_as_main\n    "__main__", mod_spec)\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 85, in _run_code\n    exec(code, run_globals)\n  File "__main__.py", line 25, in <module>\n    bootstrap_pex(__entry_point__)\n  File ".bootstrap/_pex/pex_bootstrapper.py", line 170, in bootstrap_pex\n    pex.PEX(entry_point).execute()\n  File ".bootstrap/_pex/pex.py", line 349, in execute\n    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)\n  File ".bootstrap/_pex/pex.py", line 281, in _wrap_coverage\n    return runner(*args)\n  File ".bootstrap/_pex/pex.py", line 312, in _wrap_profiling\n    return runner(*args)\n  File ".bootstrap/_pex/pex.py", line 394, in _execute\n    return self.execute_entry(self._pex_info.entry_point)\n  File ".bootstrap/_pex/pex.py", line 506, in execute_entry\n    return runner(entry_point)\n  File ".bootstrap/_pex/pex.py", line 528, in execute_pkg_resources\n    return runner()\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 87, in main\n    PantsLoader.run()\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 83, in run\n    cls.load_and_execute(entrypoint)\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 76, in load_and_execute\n    entrypoint_main()\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 474, in launch\n    PantsDaemon.Factory.create(OptionsBootstrapper.create()).run_sync()\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 407, in run_sync\n    self._run_services(self._services)\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 362, in _run_services\n    service_thread.join(self.JOIN_TIMEOUT_SECONDS)\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/threading.py", line 1036, in join\n    self._wait_for_tstate_lock(timeout=max(timeout, 0))\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock\n    elif lock.acquire(block, timeout):\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/base/exception_sink.py", line 74, in handle_sigterm\n    raise self.SignalHandledNonLocalExit(signum, \'SIGTERM\')\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/base/exception_sink.py", line 67, in __init__\n    self.traceback_lines = traceback.format_stack()\n\n\n'
                     pantsd/pantsd.log >>> 06:18:06 [WARN] Encountered error when trying to log this message: b'timestamp: 2019-04-15T06:18:06.139286\nException caught: (builtins.ValueError)\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 193, in _run_module_as_main\n    "__main__", mod_spec)\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 85, in _run_code\n    exec(code, run_globals)\n  File "__main__.py", line 25, in <module>\n    bootstrap_pex(__entry_point__)\n  File ".bootstrap/_pex/pex_bootstrapper.py", line 170, in bootstrap_pex\n    pex.PEX(entry_point).execute()\n  File ".bootstrap/_pex/pex.py", line 368, in execute\n    sys.stderr.flush()\n\nException message: I/O operation on closed file.\n\n'
                     pantsd/pantsd.log >>> 06:18:10 [INFO] setting up service <pants.pantsd.service.fs_event_service.FSEventService object at 0x7f28baac6d30>
                     pantsd/pantsd.log >>> 06:18:10 [INFO] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f28baac6f28>
                     pantsd/pantsd.log >>> 06:18:10 [INFO] watching invalidating files: set()
                     pantsd/pantsd.log >>> 06:18:10 [INFO] setting up service <pants.pantsd.service.pailgun_service.PailgunService object at 0x7f28baae0d30>
                     pantsd/pantsd.log >>> 06:18:10 [INFO] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f28baae0c88>
                     pantsd/pantsd.log >>> 06:18:10 [INFO] starting service <pants.pantsd.service.fs_event_service.FSEventService object at 0x7f28baac6d30>
                     pantsd/pantsd.log >>> 06:18:10 [INFO] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f28baac6f28>
                     pantsd/pantsd.log >>> 06:18:10 [INFO] starting service <pants.pantsd.service.pailgun_service.PailgunService object at 0x7f28baae0d30>
                     pantsd/pantsd.log >>> 06:18:10 [INFO] starting pailgun server on port 38965
                     pantsd/pantsd.log >>> 06:18:10 [INFO] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f28baae0c88>
                     pantsd/pantsd.log >>> 06:18:10 [INFO] handling pailgun request: `./pants --no-pantsrc --pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d --kill-nailguns --print-exception-stacktrace=True --pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini run testprojects/src/python/coordinated_runs:waiter -- /home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/some_magic_file`
                     pantsd/pantsd.log >>> 06:18:10 [INFO] confirmed watchman subscription: {'clock': 'c:1555309088:17746:1:2', 'subscribe': 'all_files', 'version': '4.9.1'}
                     pantsd/pantsd.log >>> 06:18:10 [INFO] enqueuing 5066 changes for subscription all_files
                     pantsd/pantsd.log >>> 06:18:10 [INFO] confirmed watchman subscription: {'clock': 'c:1555309088:17746:1:4', 'subscribe': 'pantsd_pid', 'version': '4.9.1'}
                     pantsd/pantsd.log >>> 06:18:10 [INFO] enqueuing 1 changes for subscription pantsd_pid
                     pantsd/pantsd.log >>> 06:18:10 [INFO] pailgun request completed: `./pants --no-pantsrc --pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d --kill-nailguns --print-exception-stacktrace=True --pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini run testprojects/src/python/coordinated_runs:waiter -- /home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/some_magic_file`
                     pantsd/pantsd.log --- 
                     ===============================================================
                     - BEGIN pantsd.log --------------------------------------------
                     ===============================================================
                     06:18:05 [INFO] setting up service <pants.pantsd.service.fs_event_service.FSEventService object at 0x7f651c73ecf8>
                     06:18:05 [INFO] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f651c73eef0>
                     06:18:05 [INFO] watching invalidating files: set()
                     06:18:05 [INFO] setting up service <pants.pantsd.service.pailgun_service.PailgunService object at 0x7f651c757d30>
                     06:18:05 [INFO] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f651c757c88>
                     06:18:05 [INFO] starting service <pants.pantsd.service.fs_event_service.FSEventService object at 0x7f651c73ecf8>
                     06:18:05 [INFO] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f651c73eef0>
                     06:18:05 [INFO] starting service <pants.pantsd.service.pailgun_service.PailgunService object at 0x7f651c757d30>
                     06:18:05 [INFO] starting pailgun server on port 42517
                     06:18:05 [INFO] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f651c757c88>
                     06:18:05 [INFO] handling pailgun request: `./pants --no-pantsrc --pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d --kill-nailguns --print-exception-stacktrace=True --pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini kill-pantsd`
                     06:18:05 [INFO] confirmed watchman subscription: {'clock': 'c:1555309083:17670:1:2', 'subscribe': 'all_files', 'version': '4.9.1'}
                     06:18:05 [INFO] enqueuing 5066 changes for subscription all_files
                     06:18:05 [INFO] confirmed watchman subscription: {'clock': 'c:1555309083:17670:1:4', 'subscribe': 'pantsd_pid', 'version': '4.9.1'}
                     06:18:05 [INFO] enqueuing 1 changes for subscription pantsd_pid
                     06:18:05 [INFO] pailgun request completed: `./pants --no-pantsrc --pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d --kill-nailguns --print-exception-stacktrace=True --pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini kill-pantsd`
                     06:18:06 [WARN] Encountered error when trying to log this message: b'timestamp: 2019-04-15T06:18:06.138092\nSignal 15 (SIGTERM) was raised. Exiting with failure.\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 193, in _run_module_as_main\n    "__main__", mod_spec)\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 85, in _run_code\n    exec(code, run_globals)\n  File "__main__.py", line 25, in <module>\n    bootstrap_pex(__entry_point__)\n  File ".bootstrap/_pex/pex_bootstrapper.py", line 170, in bootstrap_pex\n    pex.PEX(entry_point).execute()\n  File ".bootstrap/_pex/pex.py", line 349, in execute\n    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)\n  File ".bootstrap/_pex/pex.py", line 281, in _wrap_coverage\n    return runner(*args)\n  File ".bootstrap/_pex/pex.py", line 312, in _wrap_profiling\n    return runner(*args)\n  File ".bootstrap/_pex/pex.py", line 394, in _execute\n    return self.execute_entry(self._pex_info.entry_point)\n  File ".bootstrap/_pex/pex.py", line 506, in execute_entry\n    return runner(entry_point)\n  File ".bootstrap/_pex/pex.py", line 528, in execute_pkg_resources\n    return runner()\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 87, in main\n    PantsLoader.run()\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 83, in run\n    cls.load_and_execute(entrypoint)\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/bin/pants_loader.py", line 76, in load_and_execute\n    entrypoint_main()\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 474, in launch\n    PantsDaemon.Factory.create(OptionsBootstrapper.create()).run_sync()\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 407, in run_sync\n    self._run_services(self._services)\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/pantsd/pants_daemon.py", line 362, in _run_services\n    service_thread.join(self.JOIN_TIMEOUT_SECONDS)\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/threading.py", line 1036, in join\n    self._wait_for_tstate_lock(timeout=max(timeout, 0))\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock\n    elif lock.acquire(block, timeout):\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/base/exception_sink.py", line 74, in handle_sigterm\n    raise self.SignalHandledNonLocalExit(signum, \'SIGTERM\')\n  File "/home/travis/.pex/code/051f074d134dbfb120b31a0be8f66238f2181395/pants/base/exception_sink.py", line 67, in __init__\n    self.traceback_lines = traceback.format_stack()\n\n\n'
                     06:18:06 [WARN] Encountered error when trying to log this message: b'timestamp: 2019-04-15T06:18:06.139286\nException caught: (builtins.ValueError)\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 193, in _run_module_as_main\n    "__main__", mod_spec)\n  File "/opt/pyenv/versions/3.7.1/lib/python3.7/runpy.py", line 85, in _run_code\n    exec(code, run_globals)\n  File "__main__.py", line 25, in <module>\n    bootstrap_pex(__entry_point__)\n  File ".bootstrap/_pex/pex_bootstrapper.py", line 170, in bootstrap_pex\n    pex.PEX(entry_point).execute()\n  File ".bootstrap/_pex/pex.py", line 368, in execute\n    sys.stderr.flush()\n\nException message: I/O operation on closed file.\n\n'
                     06:18:10 [INFO] setting up service <pants.pantsd.service.fs_event_service.FSEventService object at 0x7f28baac6d30>
                     06:18:10 [INFO] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f28baac6f28>
                     06:18:10 [INFO] watching invalidating files: set()
                     06:18:10 [INFO] setting up service <pants.pantsd.service.pailgun_service.PailgunService object at 0x7f28baae0d30>
                     06:18:10 [INFO] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f28baae0c88>
                     06:18:10 [INFO] starting service <pants.pantsd.service.fs_event_service.FSEventService object at 0x7f28baac6d30>
                     06:18:10 [INFO] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f28baac6f28>
                     06:18:10 [INFO] starting service <pants.pantsd.service.pailgun_service.PailgunService object at 0x7f28baae0d30>
                     06:18:10 [INFO] starting pailgun server on port 38965
                     06:18:10 [INFO] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f28baae0c88>
                     06:18:10 [INFO] handling pailgun request: `./pants --no-pantsrc --pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d --kill-nailguns --print-exception-stacktrace=True --pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini run testprojects/src/python/coordinated_runs:waiter -- /home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/some_magic_file`
                     06:18:10 [INFO] confirmed watchman subscription: {'clock': 'c:1555309088:17746:1:2', 'subscribe': 'all_files', 'version': '4.9.1'}
                     06:18:10 [INFO] enqueuing 5066 changes for subscription all_files
                     06:18:10 [INFO] confirmed watchman subscription: {'clock': 'c:1555309088:17746:1:4', 'subscribe': 'pantsd_pid', 'version': '4.9.1'}
                     06:18:10 [INFO] enqueuing 1 changes for subscription pantsd_pid
                     06:18:10 [INFO] pailgun request completed: `./pants --no-pantsrc --pants-workdir=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d --kill-nailguns --print-exception-stacktrace=True --pants-config-files=/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/pants.ini run testprojects/src/python/coordinated_runs:waiter -- /home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.workdir.pants.d/some_magic_file`
                     ===============================================================
                     - END pantsd.log ----------------------------------------------
                     ===============================================================
                     
                     running: ./pants kill-pantsd (config={'GLOBAL': {'enable_pantsd': True, 'watchman_socket_path': '/tmp/watchman.17219.sock', 'level': 'info', 'pants_subprocessdir': '/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmp8jku5791.pants.d/.pids'}}) (extra_env={})
                     
                     completed in 2.710766077041626 seconds
                     PantsDaemonMonitor: pid is 17765 is_alive=False
                     - generated xml file: /home/travis/build/pantsbuild/pants/.pants.d/test/pytest/tests.python.pants_test.pantsd.pantsd_integration/junitxml/TEST-tests.python.pants_test.pantsd.pantsd_integration.xml -
                     ============ short test summary info =============
                     FAIL tests/python/pants_test/pantsd/test_pantsd_integration.py::TestPantsDaemonIntegration::test_pantsd_sigquit
                     1 failed, 1 passed in 40.44 seconds
jsirois added a commit to jsirois/pants that referenced this issue Apr 15, 2019
jsirois added a commit that referenced this issue Apr 16, 2019
@jsirois jsirois self-assigned this Mar 24, 2020
jsirois added a commit to jsirois/pants that referenced this issue Mar 25, 2020
This brings our skip count up from 7 python and 2 rust to 25 python and
4 rust, hopefully in the name of stability as outlined in pantsbuild#9386.

Also updates the How to Contribute docs with a pointer to pantsbuild#9386 for
the latest procedure.

Includes flaky tests from:
 pantsbuild#9313 pantsbuild#9312 pantsbuild#8679 pantsbuild#8678 pantsbuild#8520 pantsbuild#8520 pantsbuild#8520 pantsbuild#8405 pantsbuild#8193 pantsbuild#8171 pantsbuild#7836 pantsbuild#7622
 pantsbuild#7573 pantsbuild#7572 pantsbuild#7554 pantsbuild#7547 pantsbuild#7457 pantsbuild#7168 pantsbuild#7149 pantsbuild#6838 pantsbuild#6829 pantsbuild#6787 pantsbuild#6114

[ci skip-jvm-tests]  # No JVM changes made.
jsirois added a commit that referenced this issue Mar 25, 2020
This brings our skip count up from 7 python and 2 rust to 25 python and
4 rust, hopefully in the name of stability as outlined in #9386.

Also updates the How to Contribute docs with a pointer to #9386 for
the latest procedure.

Includes flaky tests from:
 #9313 #9312 #8679 #8678 #8520 #8520 #8520 #8405 #8193 #8171 #7836 #7622
 #7573 #7572 #7554 #7547 #7457 #7168 #7149 #6838 #6829 #6787 #6114
@jsirois jsirois removed their assignment Mar 25, 2020
stuhood added a commit that referenced this issue Nov 24, 2020
### Problem

In order to re-enable concurrent runs in #7654, we can no longer kill the server in order to cancel an ongoing run when the client exits (as that would kill all runs rather than just the run associated with the single cancelled client). Instead, we need to gracefully terminate ongoing runs when a `pantsd` client goes away.

### Solution

Using [new facilities](https://github.com/stuhood/nails/commits/9e805a20948a04aea6c082efbe5a9d95b7b71e31) in `nails` `0.8.0`, we detect that a client has `shutdown` the write half of the connection (or otherwise gone away, via heartbeats), and set a cancellation signal on the `Session` for the associated run. When a `Session` has been marked cancelled, the server will cleanly exit the UI loop, and interrupt any `InteractiveProcess` instances.

The client now uses Rust signal handling to intercept Ctrl+C to decide to cancel the run, and will exit the client entirely on the second signal. Because of this, the `ExceptionSink` is now manually installed, and only in locations that are running a `Scheduler` in-process: the `pantsd` server, or a no-`pantsd` local run.

### Result

In general, the server should stay up much longer, because the client no longer kills it in order to handle Ctrl+C. Fixes and unskips some tests related to Ctrl+C handling, and deletes some others. Should cause interactive processes to be consistently interruptible without killing the server.

In another PR we will use this same signal to gracefully cancel any work that was happening in the `Graph`, but since only `InteractiveProcess` or `Console` instances can interact with the UI, that change does not block landing this.

Fixes #7554, fixes #7572, fixes #7547, fixes #7457, fixes #7514.

[ci skip-build-wheels]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant