Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Buildfarmer log 2024/02/12 - 2024/02/19 #27

Closed
github-actions bot opened this issue Feb 12, 2024 · 12 comments
Closed

Buildfarmer log 2024/02/12 - 2024/02/19 #27

github-actions bot opened this issue Feb 12, 2024 · 12 comments
Labels
buildfarmer-log Log of issues in the buildfarm

Comments

@github-actions
Copy link
Contributor

Previous log #26
@github-actions github-actions bot added the buildfarmer-log Log of issues in the buildfarm label Feb 12, 2024
@github-actions GitHub Actions
Copy link
Contributor Author

Feb 12

Build regressions 
nightly_linux-rhel_debug|1816|2024-02-12 04:00:01.207000|Jenkins channel closing down (Agent disconnected)|build
Rci__nightly-performance_ubuntu_jammy_amd64|627|2024-02-11 15:52:31.552000|Packages performance_test Failed|build workspace in isolation and install
gz-fuel-tools10-debbuilder|124|2024-02-12 10:01:13.773000|Failed to compile and install build|install build dependencies from debian/control
gz-gui9-debbuilder|122|2024-02-12 10:08:27.648000|Failed to compile and install build|install build dependencies from debian/control
gz-launch8-debbuilder|116|2024-02-12 10:26:21.424000|Failed to compile and install build|install build dependencies from debian/control
gz-msgs11-debbuilder|128|2024-02-12 09:48:49.553000|Failed to compile and install build|install build dependencies from debian/control
gz-physics8-debbuilder|122|2024-02-12 10:16:31.698000|Failed to compile and install build|install build dependencies from debian/control
gz-rendering9-debbuilder|123|2024-02-12 09:53:19.652000|Failed to compile and install build|install build dependencies from debian/control
gz-sensors9-debbuilder|122|2024-02-12 10:12:57.733000|Failed to compile and install build|install build dependencies from debian/control
gz-sim9-debbuilder|120|2024-02-12 10:21:20.337000|Failed to compile and install build|install build dependencies from debian/control
gz-transport14-debbuilder|125|2024-02-12 10:04:48.333000|Failed to compile and install build|install build dependencies from debian/control
sdformat15-debbuilder|122|2024-02-12 09:57:35.639000|Failed to compile and install build|install build dependencies from debian/control
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 84.62% in 11 out of 13 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_linux_repeated
- Failing 42.86% in 6 out of 14 builds
	rcl_logging_spdlog.LoggingTest.init_old_flushing_behavior
	projectroot.test_logging_interface

Issue #3 in nightly_linux_repeated
- Failing 28.57% in 4 out of 14 builds
	ros2topic.ros2topic.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #4 in nightly_linux_humble_coverage
- Failing 18.18% in 2 out of 11 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #5 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	CameraSensorBackgroundFixture.RedBackground

Issue #6 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/MultipleServers.TwoServersMixedBlocking/0
	  (also in: "gz_sim-main-win" job)
	  (also in: "gz_sim-ign-gazebo6-win" job)

Issue #7 in gz_transport-ci-main-jammy-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Recorder_TEST.test_ran
	  (also in: "gz_transport-ci-gz-transport12-focal-amd64" job)

Issue #8 in nightly_linux-rhel_repeated
- Failing 6.67% in 1 out of 15 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)

Issue #9 in nightly_linux_humble_coverage
- Failing 9.09% in 1 out of 11 builds
	rclcpp.test_node.gtest.missing_result
	rclcpp.test_executor.gtest.missing_result
	projectroot.test.rclcpp.test_node

Issue #10 in nightly_linux_humble_coverage
- Failing 9.09% in 1 out of 11 builds
	projectroot.test.rclcpp.test_executor
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #11 in nightly_win_rep
- Failing 7.69% in 1 out of 13 builds
	test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[MultiNested] (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
	projectroot.test_publisher_subscriber__rclpy__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	projectroot.message_filters_test_latest_time_policy
	message_filters.LatestTimePolicy.Leading

@Crola1702
Copy link
Contributor

Homebrew broken bottles | Build regression | Investigating 🕵️

Reference build:

It seems they fail the test phase because of errors to load dynamic libraries, and have references to old package versions (however, I'm not sure if that's good or bad).

Log output:

gz-gui9 
+ brew test gz-gui9
==> Testing osrf/simulation/gz-gui9
==> /usr/local/opt/gz-plugin3/libexec/gz/plugin3/gz-plugin --info --plugin /usr/
Error while loading the library [/usr/local/Cellar/gz-gui9/8.999.999-0-20231016/lib/gz-gui-8/plugins/libCameraFps.dylib]: dlopen(/usr/local/Cellar/gz-gui9/8.999.999-0-20231016/lib/gz-gui-8/plugins/libCameraFps.dylib, 0x0005): tried: '/usr/local/Cellar/gz-gui9/8.999.999-0-20231016/lib/gz-gui-8/plugins/libCameraFps.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/Cellar/gz-gui9/8.999.999-0-20231016/lib/gz-gui-8/plugins/libCameraFps.dylib' (no such file), '/usr/local/Cellar/gz-gui9/8.999.999-0-20231016/lib/gz-gui-8/plugins/libCameraFps.dylib' (no such file)
Error: osrf/simulation/gz-gui9: failed
An exception occurred within a child process:
  Minitest::Assertion: Error while loading the library
/usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/minitest-5.21.2/lib/minitest/assertions.rb:183:in `assert'
/usr/local/Homebrew/Library/Taps/osrf/homebrew-simulation/Formula/gz-gui9.rb:52:in `block (2 levels) in <class:GzGui9>'
/usr/local/Homebrew/Library/Taps/osrf/homebrew-simulation/Formula/gz-gui9.rb:42:in `each'
/usr/local/Homebrew/Library/Taps/osrf/homebrew-simulation/Formula/gz-gui9.rb:42:in `block in <class:GzGui9>'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2605:in `block (3 levels) in run_test'
/usr/local/Homebrew/Library/Homebrew/extend/kernel.rb:493:in `with_env'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2604:in `block (2 levels) in run_test'
/usr/local/Homebrew/Library/Homebrew/formula.rb:1058:in `with_logging'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2603:in `block in run_test'
/usr/local/Homebrew/Library/Homebrew/mktemp.rb:75:in `block in run'
/usr/local/Homebrew/Library/Homebrew/mktemp.rb:75:in `chdir'
/usr/local/Homebrew/Library/Homebrew/mktemp.rb:75:in `run'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2897:in `mktemp'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2597:in `run_test'
/usr/local/Homebrew/Library/Homebrew/test.rb:46:in `block in <main>'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:36:in `block in catch'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:36:in `catch'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:36:in `catch'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:123:in `timeout'
/usr/local/Homebrew/Library/Homebrew/test.rb:50:in `<main>'
gz-launch8 
+ brew test gz-launch8
==> Testing osrf/simulation/gz-launch8
==> /usr/local/Cellar/gz-launch8/7.999.999-0-20231016/lib/gz/launch8/gz-launch
==> /usr/local/opt/gz-plugin3/libexec/gz/plugin3/gz-plugin --info --plugin /usr/
Error while loading the library [/usr/local/Cellar/gz-launch8/7.999.999-0-20231016/lib/gz-launch-7/plugins/libgz-launch-joytotwist.dylib]: dlopen(/usr/local/Cellar/gz-launch8/7.999.999-0-20231016/lib/gz-launch-7/plugins/libgz-launch-joytotwist.dylib, 0x0005): tried: '/usr/local/Cellar/gz-launch8/7.999.999-0-20231016/lib/gz-launch-7/plugins/libgz-launch-joytotwist.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/Cellar/gz-launch8/7.999.999-0-20231016/lib/gz-launch-7/plugins/libgz-launch-joytotwist.dylib' (no such file), '/usr/local/Cellar/gz-launch8/7.999.999-0-20231016/lib/gz-launch-7/plugins/libgz-launch-joytotwist.dylib' (no such file)
Error: osrf/simulation/gz-launch8: failed
An exception occurred within a child process:
  Minitest::Assertion: Error while loading the library
/usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/minitest-5.21.2/lib/minitest/assertions.rb:183:in `assert'
/usr/local/Homebrew/Library/Taps/osrf/homebrew-simulation/Formula/gz-launch8.rb:58:in `block (2 levels) in <class:GzLaunch8>'
/usr/local/Homebrew/Library/Taps/osrf/homebrew-simulation/Formula/gz-launch8.rb:48:in `each'
/usr/local/Homebrew/Library/Taps/osrf/homebrew-simulation/Formula/gz-launch8.rb:48:in `block in <class:GzLaunch8>'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2605:in `block (3 levels) in run_test'
/usr/local/Homebrew/Library/Homebrew/extend/kernel.rb:493:in `with_env'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2604:in `block (2 levels) in run_test'
/usr/local/Homebrew/Library/Homebrew/formula.rb:1058:in `with_logging'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2603:in `block in run_test'
/usr/local/Homebrew/Library/Homebrew/mktemp.rb:75:in `block in run'
/usr/local/Homebrew/Library/Homebrew/mktemp.rb:75:in `chdir'
/usr/local/Homebrew/Library/Homebrew/mktemp.rb:75:in `run'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2897:in `mktemp'
/usr/local/Homebrew/Library/Homebrew/formula.rb:2597:in `run_test'
/usr/local/Homebrew/Library/Homebrew/test.rb:46:in `block in <main>'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:36:in `block in catch'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:36:in `catch'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:36:in `catch'
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:123:in `timeout'
/usr/local/Homebrew/Library/Homebrew/test.rb:50:in `<main>'
Build step 'Execute shell' marked build as failure

@Crola1702
Copy link
Contributor

Crola1702 commented Feb 12, 2024 •
edited
Loading

ros_gz_{garden,harmonic}_{humble,iron} | Build regressions | Investigating 🕵️

Reference builds:

Log output:

# BEGIN SECTION: ros_gz talker/listener
+ sleep 1
+ ros2 run ros_gz_bridge parameter_bridge /chatter@std_msgs/msg/String@gz.msgs.StringMsg
+ sleep 1
+ ros2 topic echo /chatter
+ sleep 1
+ gz topic -t /chatter -m gz.msgs.StringMsg -p 'data:"Hello"'
+ grep -v Hello /tmp/echo_chatter
+ echo 'chatter log file does not contain the expected Hello string'
chatter log file does not contain the expected Hello string
+ exit 1
Build step 'Execute shell' marked build as failure

It doesn't seem to be changes in the repositories

I'm not familiar with these wrappers work, so I'll report to the team.

@scpeters scpeters mentioned this issue Feb 12, 2024
Merged
@scpeters
Copy link
Member

Homebrew broken bottles | Build regression | Investigating 🕵️

Reference build:

It seems they fail the test phase because of errors to load dynamic libraries, and have references to old package versions (however, I'm not sure if that's good or bad).

Log output:

gz-gui9
gz-launch8

There were errors in the formulae, which had been previously failing for different reasons. The file paths will be fixed by osrf/homebrew-simulation#2557 though there is at least one more issue with gz-gui9 that I will fix separately

@github-actions GitHub Actions
Copy link
Contributor Author

Feb 13

Build regressions 
nightly_linux_humble_coverage|708|2024-02-13 05:20:41.801000|Packages rosbag2_transport  Failed|build
packaging_windows|3146|2024-02-13 07:09:30.035000|Early fail|No last section
packaging_windows_debug|2106|2024-02-13 07:12:35.445000|Early fail|No last section
Rci__nightly-fastrtps_ubuntu_jammy_amd64|732|2024-02-12 14:21:13.805000|Unknown Reason|import repositories
Rci__nightly-performance_ubuntu_jammy_amd64|628|2024-02-12 20:00:13.634000|Packages performance_test Failed|build workspace in isolation and install
gz-gui9-debbuilder|125|2024-02-12 16:05:29.116000|Failed to compile and install build|install build dependencies from debian/control
gz-gui9-debbuilder|124|2024-02-12 15:42:51.197000|Failed to compile and install build|install build dependencies from debian/control
gz-rendering9-debbuilder|124|2024-02-12 16:10:00.699000|Failed to compile and install build|install build dependencies from debian/control
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 83.33% in 10 out of 12 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_linux_repeated
- Failing 42.86% in 6 out of 14 builds
	rcl_logging_spdlog.LoggingTest.init_old_flushing_behavior
	projectroot.test_logging_interface

Issue #3 in nightly_linux_repeated
- Failing 28.57% in 4 out of 14 builds
	ros2topic.ros2topic.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #4 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 12.5% in 1 out of 8 builds
	UNIT_Gui_clean_exit_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/1886 - CLOSED)

Issue #5 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	CameraSensorBackgroundFixture.RedBackground

Issue #6 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/MultipleServers.TwoServersMixedBlocking/0
	  (also in: "gz_sim-main-win" job)

Issue #7 in gz_transport-ci-main-jammy-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Recorder_TEST.test_ran
	  (also in: "gz_transport-ci-gz-transport12-focal-amd64" job)

Issue #8 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #9 in nightly_win_rep
- Failing 8.33% in 1 out of 12 builds
	test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[MultiNested] (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]
	projectroot.test_publisher_subscriber__rclpy__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	projectroot.message_filters_test_latest_time_policy
	message_filters.LatestTimePolicy.Leading

@github-actions GitHub Actions
Copy link
Contributor Author

Feb 14

Build regressions 
nightly_linux-aarch64_release|2638|2024-02-14 04:00:01.444000|Jenkins channel closing down (Agent disconnected)|build
Rci__nightly-performance_ubuntu_jammy_amd64|629|2024-02-13 21:54:49.110000|Packages performance_test Failed|build workspace in isolation and install
gz-common6-debbuilder|126|2024-02-14 10:05:03.178000|Failed to fetch some archives|see Dockerfile
gz-launch8-debbuilder|118|2024-02-13 17:16:47.310000|Failed to compile and install build|install build dependencies from debian/control
gz-physics8-debbuilder|125|2024-02-13 17:12:35.297000|Unknown Reason|see Dockerfile
gz-plugin3-debbuilder|128|2024-02-14 10:07:25.764000|Failed to fetch some archives|see Dockerfile
gz-sensors9-debbuilder|124|2024-02-13 16:40:49.206000|Unknown Reason|see Dockerfile
gz-sim9-debbuilder|122|2024-02-13 17:12:41.973000|Failed to compile and install build|install build dependencies from debian/control
gz-tools2-debbuilder|1406|2024-02-14 09:13:10.945000|Failed to compile and install build|install build dependencies from debian/control
gz-tools2-debbuilder|1403|2024-02-13 14:41:50.621000|Failed to compile and install build|install build dependencies from debian/control
gz-tools2-debbuilder|1401|2024-02-13 13:10:50.267000|Early fail|No last section
gz_sim-8-win|29|2024-02-13 14:54:16.033000|Jenkins channel closing down (Agent disconnected)|colcon compilation without test for dependencies of gz-sim8
gz_sim-8-win|28|2024-02-13 14:06:41.719000|Jenkins channel closing down (Agent disconnected)|colcon compilation with tests for gz-sim8
gz_sim-ci-gz-sim8-homebrew-amd64|31|2024-02-13 14:06:40.617000|Failed to compile and install gz-sim8|install gz-sim8 dependencies
gz_launch-ci-gz-launch6-homebrew-amd64|20|2024-02-13 14:06:40.607000|Failed to compile and install gz-launch6|install gz-launch6 dependencies
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 83.33% in 10 out of 12 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_win_rep
- Failing 33.33% in 4 out of 12 builds
	projectroot.test.rclcpp.test_timers_manager

Issue #3 in nightly_win_rep
- Failing 25.0% in 3 out of 12 builds
	rclcpp.TestTimersManager.check_one_timer_cancel_doesnt_affect_other_timers

Issue #4 in nightly_win_rep
- Failing 25.0% in 3 out of 12 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_win_rel" job)

Issue #5 in nightly_win_rep
- Failing 16.67% in 2 out of 12 builds
	test_cli_remapping.TestCLIRemappingAfterShutdown.test_processes_finished_gracefully[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]
	  (also in: "nightly_win_rel" job)

Issue #6 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 12.5% in 1 out of 8 builds
	UNIT_Gui_clean_exit_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/1886 - CLOSED)

Issue #7 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	CameraSensorBackgroundFixture.RedBackground

Issue #8 in gz_sim-ign-gazebo6-win
- Failing 33.33% in 1 out of 3 builds
	ServerRepeat/MultipleServers.TwoServersMixedBlocking/0
	  (also in: "gz_sim-main-win" job)

Issue #9 in gz_transport-ci-gz-transport13-homebrew-amd64
- Failing 20.0% in 1 out of 5 builds
	gzTest.TopicEchoNum

Issue #10 in nightly_win_rep
- Failing 8.33% in 1 out of 12 builds
	test_cli_remapping.TestCLIRemapping.test_namespace_replacement[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]

@github-actions GitHub Actions
Copy link
Contributor Author

Feb 15

Build regressions 
gz-launch8-debbuilder|121|2024-02-15 02:49:12.100000|Failed to compile and install build|install build dependencies from debian/control
gz-physics8-debbuilder|127|2024-02-15 02:39:51.739000|Failed to compile and install build|install build dependencies from debian/control
gz-sim9-debbuilder|125|2024-02-15 02:43:24.543000|Failed to compile and install build|install build dependencies from debian/control
gz-tools2-debbuilder|1409|2024-02-15 09:40:36.600000|Failed to compile and install build|install build dependencies from debian/control
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_linux_repeated
- Failing 28.57% in 4 out of 14 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #2 in gz_sim-ci-ign-gazebo6-homebrew-amd64
- Failing 50.0% in 2 out of 4 builds
	ServerRepeat/OdometryPublisherTest.GaussianNoiseTest/0

Issue #3 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 14.29% in 1 out of 7 builds
	UNIT_Gui_clean_exit_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/1886 - CLOSED)

Issue #4 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	CameraSensorBackgroundFixture.RedBackground

Issue #5 in gz_transport-ci-gz-transport13-homebrew-amd64
- Failing 20.0% in 1 out of 5 builds
	gzTest.TopicEchoNum

Issue #6 in gz_transport-ci-ign-transport11-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_twoProcsPubSub.test_ran

Issue #7 in nightly_win_deb
- Failing 50.0% in 1 out of 2 builds
	launch_testing_ros.test.examples.talker_listener_launch_test.talker_listener_launch_test

@github-actions GitHub Actions
Copy link
Contributor Author

Feb 16

Build regressions 
Hci__nightly-performance_ubuntu_jammy_amd64|204|2024-02-16 01:18:53.085000|Packages performance_test Failed|build workspace in isolation and install
gz-tools2-debbuilder|1411|2024-02-16 09:14:12.247000|Failed to compile and install build|install build dependencies from debian/control
gz_physics-ign-physics2-win|22|2024-02-15 21:10:36.572000|Early fail|No last section
gz_physics-ign-physics2-win|21|2024-02-15 21:06:02.735000|Early fail|No last section
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 85.71% in 12 out of 14 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_win_rep
- Failing 50.0% in 7 out of 14 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp

Issue #3 in nightly_win_rep
- Failing 42.86% in 6 out of 14 builds
	projectroot.test.rclcpp.test_timers_manager

Issue #4 in nightly_linux_repeated
- Failing 35.71% in 5 out of 14 builds
	rcl_logging_spdlog.LoggingTest.init_old_flushing_behavior
	projectroot.test_logging_interface

Issue #5 in nightly_win_rep
- Failing 35.71% in 5 out of 14 builds
	test_rclcpp.TestNNodes.test_10_nodes

Issue #6 in nightly_win_rep
- Failing 35.71% in 5 out of 14 builds
	rclcpp.TestTimersManager.check_one_timer_cancel_doesnt_affect_other_timers

Issue #7 in gz_sim-ci-ign-gazebo6-homebrew-amd64
- Failing 50.0% in 2 out of 4 builds
	ServerRepeat/OdometryPublisherTest.GaussianNoiseTest/0

Issue #8 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #9 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 16.67% in 1 out of 6 builds
	UNIT_Gui_clean_exit_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/1886 - CLOSED)

Issue #10 in gz_transport-ci-gz-transport13-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	gzTest.TopicEchoNum

Issue #11 in gz_transport-ci-ign-transport11-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_twoProcsPubSub.test_ran

Issue #12 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	rclcpp.test_executors_timer_cancel_behavior.gtest.missing_result
	  (also in: "nightly_linux_repeated" job)

Issue #13 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[BoundedPlainSequences] (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	rclcpp.TestTimerCancelBehavior/StaticSingleThreadedExecutor.testBothTimerCancelThenResetT2Behavior
	projectroot.test_tutorial_content_filtering_publisher_content_filtering_subscriber__rmw_fastrtps_cpp
	projectroot.test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp
	projectroot.test_publisher_subscriber__rclcpp__rmw_fastrtps_cpp

Issue #14 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	projectroot.test.rclcpp.test_executors_timer_cancel_behavior
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #15 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	projectroot.test_tutorial_use_logger_service__rmw_fastrtps_cpp
	demo_nodes_cpp.TestExecutablesTutorialAfterShutdown.test_last_process_exit_code
	  (also in: "nightly_linux-aarch64_repeated" job)

@Crola1702
Copy link
Contributor

Crola1702 commented Feb 16, 2024 •
edited
Loading

Humble performance | Build regression | Investigating 🕵️

Reference build: https://build.ros2.org/view/Hci/job/Hci__nightly-performance_ubuntu_jammy_amd64/204/

Log output:

00:04:34.899 /tmp/ws2/build_isolated/performance_test/src/idlgen/fast_rtps/gen/fast_rtps/Header_PubSubTypes.cxx:23:10: fatal error: fastdds/rtps/common/CdrSerialization.hpp: No such file or directory
00:04:34.899    23 | #include <fastdds/rtps/common/CdrSerialization.hpp>
00:04:34.899       |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
00:04:34.899 compilation terminated.

@clalancette
Copy link
Contributor

Reference build: https://build.ros2.org/view/Hci/job/Hci__nightly-performance_ubuntu_jammy_amd64/204/

Arg. That's probably because of ros2/performance_test#42 . I forgot that this same branch was used for Rolling and Humble. We'll have to rethink that, and maybe make a new branch.

@github-actions GitHub Actions
Copy link
Contributor Author

Feb 17

Build regressions 
nightly_linux_humble_coverage|712|2024-02-17 05:49:17.040000|Number of failed test exceeds the threshold|ccache stats (after)
Ici__nightly-performance_ubuntu_jammy_amd64|83|2024-02-17 03:14:25.763000|Packages performance_test Failed|build workspace in isolation and install
gz-launch8-debbuilder|124|2024-02-16 11:43:44.741000|Failed to compile and install build|install build dependencies from debian/control
gz-sim9-debbuilder|128|2024-02-16 11:38:58.867000|Failed to compile and install build|install build dependencies from debian/control
gz-tools2-debbuilder|1415|2024-02-17 09:13:03.696000|Failed to compile and install build|install build dependencies from debian/control
gz_sensors-ci-gz-sensors7-homebrew-amd64|21|2024-02-17 10:06:03.601000|Failed to compile and install gz-sensors7|install gz-sensors7 dependencies
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 84.62% in 11 out of 13 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_win_rep
- Failing 53.85% in 7 out of 13 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp

Issue #3 in nightly_win_rep
- Failing 38.46% in 5 out of 13 builds
	test_rclcpp.TestNNodes.test_10_nodes

Issue #4 in nightly_win_rep
- Failing 38.46% in 5 out of 13 builds
	projectroot.test.rclcpp.test_timers_manager

Issue #5 in nightly_win_rep
- Failing 30.77% in 4 out of 13 builds
	rclcpp.TestTimersManager.check_one_timer_cancel_doesnt_affect_other_timers

Issue #6 in nightly_win_rep
- Failing 23.08% in 3 out of 13 builds
	projectroot.test.rclcpp.test_executors_timer_cancel_behavior
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #7 in gz_sim-ci-ign-gazebo6-homebrew-amd64
- Failing 50.0% in 2 out of 4 builds
	ServerRepeat/OdometryPublisherTest.GaussianNoiseTest/0

Issue #8 in nightly_win_rep
- Failing 15.38% in 2 out of 13 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #9 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 20.0% in 1 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #10 in gz_sim-main-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #11 in gz_transport-ci-gz-transport13-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	gzTest.TopicEchoNum

Issue #12 in gz_transport-ci-ign-transport11-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_twoProcsPubSub.test_ran

Issue #13 in nightly_win_rep
- Failing 7.69% in 1 out of 13 builds
	test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[BoundedPlainSequences] (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	rclcpp.TestTimerCancelBehavior/StaticSingleThreadedExecutor.testBothTimerCancelThenResetT2Behavior
	projectroot.test_tutorial_content_filtering_publisher_content_filtering_subscriber__rmw_fastrtps_cpp
	projectroot.test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp
	projectroot.test_publisher_subscriber__rclcpp__rmw_fastrtps_cpp

Issue #14 in nightly_win_rep
- Failing 7.69% in 1 out of 13 builds
	projectroot.test_tutorial_use_logger_service__rmw_fastrtps_cpp
	demo_nodes_cpp.TestExecutablesTutorialAfterShutdown.test_last_process_exit_code
	  (also in: "nightly_linux-aarch64_repeated" job)

@github-actions GitHub Actions
Copy link
Contributor Author

Feb 18

Build regressions 
nightly_linux-rhel_repeated|1807|2024-02-18 04:00:01.183000|Packages action_tutorials_cpp action_tutorials_py camera_calibration_parsers camera_info_manager composition demo_nodes_cpp demo_nodes_cpp_native demo_nodes_py dummy_map_server dummy_robot_bringup dummy_sensors examples_rclcpp_async_client examples_rclcpp_cbg_executor examples_rclcpp_minimal_action_client examples_rclcpp_minimal_action_server examples_rclcpp_minimal_client examples_rclcpp_minimal_composition examples_rclcpp_minimal_publisher examples_rclcpp_minimal_service examples_rclcpp_minimal_subscriber examples_rclcpp_minimal_timer examples_rclcpp_multithreaded_executor examples_rclcpp_wait_set examples_rclpy_executors examples_rclpy_guard_conditions examples_rclpy_minimal_action_client examples_rclpy_minimal_action_server examples_rclpy_minimal_client examples_rclpy_minimal_publisher examples_rclpy_minimal_service examples_rclpy_minimal_subscriber examples_rclpy_pointcloud_publisher examples_tf2_py geometry2 image_common image_tools image_transport interactive_markers intra_process_demo laser_geometry launch_ros launch_testing_examples launch_testing_ros lifecycle lifecycle_py logging_demo message_filters pendulum_control point_cloud_transport point_cloud_transport_py quality_of_service_demo_cpp quality_of_service_demo_py rclcpp_action rclcpp_components rclcpp_lifecycle robot_state_publisher ros2action ros2bag ros2cli ros2cli_common_extensions ros2component ros2doctor ros2interface ros2launch ros2lifecycle ros2lifecycle_test_fixtures ros2multicast ros2node ros2param ros2pkg ros2run ros2service ros2test ros2topic ros2trace ros_testing rosbag2 rosbag2_compression rosbag2_compression_zstd rosbag2_cpp rosbag2_examples_cpp rosbag2_examples_py rosbag2_performance_benchmarking rosbag2_py rosbag2_storage rosbag2_storage_default_plugins rosbag2_storage_mcap rosbag2_storage_sqlite3 rosbag2_test_common rosbag2_tests rosbag2_transport rqt rqt_action rqt_console rqt_graph rqt_gui rqt_gui_cpp rqt_gui_py rqt_msg rqt_plot rqt_publisher rqt_py_common rqt_py_console rqt_reconfigure rqt_service_caller rqt_shell rqt_srv rqt_topic rviz2 rviz_common rviz_default_plugins rviz_visual_testing_framework sros2 sros2_cmake test_cli test_cli_remapping test_communication test_launch_ros test_quality_of_service test_rclcpp test_ros2trace test_security test_tf2 test_tracetools test_tracetools_launch tf2_bullet tf2_eigen tf2_geometry_msgs tf2_kdl tf2_py tf2_ros tf2_ros_py tf2_sensor_msgs tf2_tools tlsf_cpp topic_monitor topic_statistics_demo tracetools_launch tracetools_test turtlesim  Failed|test
gz-launch8-debbuilder|126|2024-02-17 11:48:05.043000|Failed to compile and install build|install build dependencies from debian/control
gz-physics8-debbuilder|133|2024-02-17 11:40:40.078000|Failed to compile and install build|install build dependencies from debian/control
gz-sim9-debbuilder|131|2024-02-17 11:44:12.619000|Failed to compile and install build|install build dependencies from debian/control
gz-tools2-debbuilder|1417|2024-02-18 09:13:03.263000|Failed to compile and install build|install build dependencies from debian/control
gz_launch-ci-gz-launch7-homebrew-amd64|23|2024-02-18 08:06:02.727000|Failed to compile and install gz-launch7|install gz-launch7 dependencies
gz_launch-ci-ign-launch5-homebrew-amd64|21|2024-02-18 08:06:02.736000|Failed to compile and install gz-launch5|install gz-launch5 dependencies
gz_launch-ci-ign-launch2-homebrew-amd64|21|2024-02-17 23:06:04.358000|Failed to compile and install gz-launch2|install gz-launch2 dependencies
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 84.62% in 11 out of 13 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_win_rep
- Failing 46.15% in 6 out of 13 builds
	projectroot.test.rclcpp.test_timers_manager

Issue #3 in nightly_win_rep
- Failing 38.46% in 5 out of 13 builds
	rclcpp.TestTimersManager.check_one_timer_cancel_doesnt_affect_other_timers

Issue #4 in nightly_win_rep
- Failing 30.77% in 4 out of 13 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_win_rel" job)

Issue #5 in nightly_linux_repeated
- Failing 20.0% in 3 out of 15 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #6 in nightly_linux_repeated
- Failing 20.0% in 3 out of 15 builds
	ros2topic.ros2topic.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #7 in nightly_linux_repeated
- Failing 20.0% in 3 out of 15 builds
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #8 in nightly_win_rep
- Failing 23.08% in 3 out of 13 builds
	test_cli_remapping.TestCLIRemappingAfterShutdown.test_processes_finished_gracefully[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]
	  (also in: "nightly_win_rel" job)

Issue #9 in nightly_win_rep
- Failing 15.38% in 2 out of 13 builds
	test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
	  (also in: "nightly_linux_repeated" job)

Issue #10 in nightly_win_rep
- Failing 15.38% in 2 out of 13 builds
	test_cli_remapping.TestCLIRemapping.test_topic_replacement[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]

Issue #11 in gz_sim-ci-ign-gazebo6-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	ServerRepeat/OdometryPublisherTest.GaussianNoiseTest/0

Issue #12 in gz_sim-main-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #13 in gz_transport-ci-gz-transport13-homebrew-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_Recorder_TEST.test_ran
	  (also in: "gz_transport-ci-main-jammy-amd64" job)

Issue #14 in gz_transport-ci-gz-transport13-homebrew-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_Node_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering6-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)
	  (also in: "gz_rendering-ci-ign-rendering3-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-rendering/issues/847 - CLOSED | https://github.com/gazebosim/gz-rendering/issues/847 - OPEN)

Issue #15 in gz_transport-ci-gz-transport13-homebrew-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_result_TEST.test_ran
	UNIT_gz_src_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-fuel-tools/issues/307 - CLOSED | https://github.com/gazebosim/gz-fuel-tools/issues/315 - CLOSED)
	UNIT_gz_TEST.test_ran
	UNIT_Uuid_TEST.test_ran
	UNIT_Utils_TEST.test_ran
	UNIT_TopicUtils_TEST.test_ran
	UNIT_TopicStorage_TEST.test_ran
	UNIT_TopicStatistics_TEST.test_ran
	UNIT_SubscribeOptions_TEST.test_ran
	UNIT_SqlStatement_TEST.test_ran
	UNIT_Registry_TEST.test_ran
	UNIT_QueryOptions_TEST.test_ran
	UNIT_QualifiedTime_TEST.test_ran
	UNIT_Publisher_TEST.test_ran
	UNIT_Playback_TEST.test_ran
	UNIT_ParamCommandAPI_TEST.test_ran
	UNIT_NodeOptions_TEST.test_ran
	UNIT_NetUtils_TEST.test_ran
	UNIT_Message_TEST.test_ran
	UNIT_MessageInfo_TEST.test_ran
	UNIT_Log_TEST.test_ran
	UNIT_LogCommandAPI_TEST.test_ran
	UNIT_Helpers_TEST.test_ran
	UNIT_HandlerStorage_TEST.test_ran
	UNIT_Discovery_TEST.test_ran
	UNIT_Descriptor_TEST.test_ran
	UNIT_Clock_TEST.test_ran
	UNIT_Client_TEST.test_ran
	UNIT_CIface_TEST.test_ran
	UNIT_Batch_TEST.test_ran
	UNIT_AdvertiseOptions_TEST.test_ran
	INTEGRATION_twoProcsSrvCallWithoutOutputStress.test_ran
	INTEGRATION_twoProcsSrvCallWithoutOutput.test_ran
	INTEGRATION_twoProcsSrvCallWithoutInputSync1.test_ran
	INTEGRATION_twoProcsSrvCallWithoutInputStress.test_ran
	INTEGRATION_twoProcsSrvCallWithoutInput.test_ran
	INTEGRATION_twoProcsSrvCallSync1.test_ran
	INTEGRATION_twoProcsSrvCallStress.test_ran
	INTEGRATION_twoProcsSrvCall.test_ran
	INTEGRATION_statistics.test_ran
	INTEGRATION_scopedTopic.test_ran
	INTEGRATION_recorder.test_ran
	INTEGRATION_query.test_ran
	INTEGRATION_playback.test_ran
	INTEGRATION_callback_scope_TEST.test_ran
	INTEGRATION_authPubSub.test_ran
	.gz-transport.python.test.requester_TEST
	.gz-transport.python.test.pubSub_TEST
	.gz-transport.python.test.options_TEST

Issue #16 in gz_transport-ci-gz-transport13-homebrew-amd64
- Failing 20.0% in 1 out of 5 builds
	INTEGRATION_twoProcsPubSub.test_ran
	  (also in: "gz_transport-ci-ign-transport11-homebrew-amd64" job)
	  (also in: "gz_transport-ci-gz-transport13-homebrew-amd64" job)

Issue #17 in gz_transport-main-win
- Failing 25.0% in 1 out of 4 builds
	playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-13-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)

Issue #18 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 20.0% in 1 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Ici__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #19 in nightly_win_rep
- Failing 7.69% in 1 out of 13 builds
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_fastrtps_cpp__non_secure_comm_0

@github-actions github-actions bot mentioned this issue Feb 19, 2024
Closed
This was referenced Apr 11, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
buildfarmer-log Log of issues in the buildfarm
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@clalancette @scpeters @Crola1702