Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Buildfarmer log 2024/01/15 - 2024/01/22 #23

Closed
github-actions bot opened this issue Jan 15, 2024 · 8 comments
Closed

Buildfarmer log 2024/01/15 - 2024/01/22 #23

github-actions bot opened this issue Jan 15, 2024 · 8 comments
Labels
buildfarmer-log Log of issues in the buildfarm

Comments

@github-actions
Copy link
Contributor

Previous log #21
@github-actions github-actions bot added the buildfarmer-log Log of issues in the buildfarm label Jan 15, 2024
@github-actions GitHub Actions
Copy link
Contributor Author

Jan 15

Build regressions 
packaging_linux|3297|2024-01-15 07:52:59.385000|Jenkins channel closing down (Agent disconnected)|build underlay packages
nightly_win_deb|2973|2024-01-15 04:00:01.209000|Number of failed test exceeds the threshold|test
gz-msgs11-debbuilder|92|2024-01-15 09:47:58.328000|Error in configuration dependencies|create deb packages
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 30.77% in 4 out of 13 builds
	rosidl_typesupport_introspection_tests.MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage
	projectroot.test_multi_nested_message_introspection
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #2 in gz_sim-main-win
- Failing 40.0% in 2 out of 5 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #3 in nightly_linux_repeated
- Failing 12.5% in 2 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #4 in nightly_win_rep
- Failing 15.38% in 2 out of 13 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp

Issue #5 in gz_sim-7-win
- Failing 20.0% in 1 out of 5 builds
	INTEGRATION_multiple_servers.test_ran
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-7-win" job)

Issue #6 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp
	projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp

@github-actions GitHub Actions
Copy link
Contributor Author

Jan 16

Build regressions 
nightly_win_deb|2974|2024-01-16 04:00:01.176000|Number of failed test exceeds the threshold|test
gz-msgs11-debbuilder|93|2024-01-16 09:49:02.589000|Error in configuration dependencies|create deb packages
gz_msgs-10-win|14|2024-01-16 05:54:09.560000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 35.71% in 5 out of 14 builds
	rosidl_typesupport_introspection_tests.MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage
	projectroot.test_multi_nested_message_introspection
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #2 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	ros2topic.ros2topic.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #3 in nightly_linux-rhel_repeated
- Failing 12.5% in 2 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp
	projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)

Issue #4 in gz_math-7-win
- Failing 25.0% in 1 out of 4 builds
	Stopwatch.StartStopReset

Issue #5 in gz_sim-7-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_multiple_servers.test_ran
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-7-win" job)

Issue #6 in gz_sim-main-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #7 in nightly_linux_repeated
- Failing 6.25% in 1 out of 16 builds
	test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[Strings] (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	projectroot.test_publisher_subscriber__rclcpp__rmw_fastrtps_cpp__rmw_cyclonedds_cpp

Issue #8 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
	projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__non_secure_comm_2

@Crola1702
Copy link
Contributor

Crola1702 commented Jan 16, 2024 •
edited
Loading

Windows repeated | Test regression | Investigating 🕵️

Reference build: https://ci.ros2.org/view/nightly/job/nightly_win_rep/3235/

Test regressions:

Log output: 
C:\ci\ws\src\ros2\rosidl\rosidl_typesupport_introspection_tests\test\test_multi_nested_message_introspection.cpp:214
Expected equality of these values:
  item_reference
    Which is: 736-byte object <50-01 0E-15 A1-01 00-00 C0-FB E1-AB A1-01 00-00 0B-00 00-00 00-00 00-00 FC-A8 F7-E1 00-00 00-00 3D-00 00-00 00-00 00-00 D0-8B 11-15 A1-01 00-00 00-00 00-00 00-00 00-E0 00-00 00-00 00-00 00-00 ... 01-00 00-00 00-00 00-00 20-EA 11-15 A1-01 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 20-E6 11-15 A1-01 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 65-00 00-00 FF-7F 00-00>
  message.array_of_arrays[i]
    Which is: 736-byte object <50-01 0E-15 A1-01 00-00 C0-78 11-15 A1-01 00-00 0B-00 00-00 00-00 00-00 FC-A8 F7-E1 00-00 00-00 3D-00 00-00 00-00 00-00 D0-8B 11-15 A1-01 00-00 00-00 00-00 00-00 00-E0 00-00 00-00 00-00 00-00 ... 01-00 00-00 00-00 00-00 20-54 10-15 A1-01 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 A0-54 10-15 A1-01 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 65-00 00-00 00-00 00-00>

C:\ci\ws\src\ros2\rosidl\rosidl_typesupport_introspection_tests\test\test_multi_nested_message_introspection.cpp:214
Expected equality of these values:
  item_reference
    Which is: 736-byte object <00-00 00-00 00-00 00-00 0A-FB E1-AB 00-2C 00-80 6F-12 83-C0 CA-21 09-40 00-00 00-00 00-00 00-00 6F-12 83-C0 CA-21 09-C0 04-3A 31-E4 00-2D 00-80 00-00 00-00 FF-FF FF-7F 00-00 00-80 00-00 00-00 ... 01-00 00-00 00-00 00-00 C0-E8 11-15 A1-01 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 A0-E6 11-15 A1-01 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 00-00 00-00 FF-7F 00-00>
  message.array_of_arrays[i]
    Which is: 736-byte object <00-00 00-00 00-00 00-00 0A-3A 33-E4 00-2C 00-80 6F-12 83-C0 CA-21 09-40 00-00 00-00 00-00 00-00 6F-12 83-C0 CA-21 09-C0 04-3A 31-E4 00-2D 00-80 00-00 00-00 FF-FF FF-7F 00-00 00-80 00-00 00-00 ... 01-00 00-00 00-00 00-00 60-4E 10-15 A1-01 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 60-52 10-15 A1-01 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>

C:\ci\ws\src\ros2\rosidl\rosidl_typesupport_introspection_tests\test\test_multi_nested_message_introspection.cpp:256
Expected equality of these values:
  message
    Which is: 6864-byte object <50-01 0E-15 A1-01 00-00 C0-78 11-15 A1-01 00-00 0B-00 00-00 00-00 00-00 FC-A8 F7-E1 00-00 00-00 3D-00 00-00 00-00 00-00 D0-8B 11-15 A1-01 00-00 00-00 00-00 00-00 00-E0 00-00 00-00 00-00 00-00 ... 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 80-BE 11-15 A1-01 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 80-C1 11-15 A1-01 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00>
  message_copy
    Which is: 6864-byte object <50-01 0E-15 A1-01 00-00 C0-FB E1-AB A1-01 00-00 0B-00 00-00 00-00 00-00 FC-A8 F7-E1 00-00 00-00 3D-00 00-00 00-00 00-00 D0-8B 11-15 A1-01 00-00 00-00 00-00 00-00 00-E0 00-00 00-00 00-00 00-00 ... 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 20-20 12-15 A1-01 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 50-36 12-15 A1-01 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00>

Flaky ratio:

job_name last_fail first_fail build_count failure_count failure_percentage
nightly_win_rep 2024-01-16 2023-12-19 28 7 25.0
nightly_linux-aarch64_repeated 2024-01-04 2024-01-04 31 1 3.23

aarch64 repeated reference build

First time it was seen was on 2022-02-03, so it's been failing with a 25-30% rate since then.

@github-actions GitHub Actions
Copy link
Contributor Author

Jan 17

Build regressions 
nightly_win_deb|2975|2024-01-17 04:00:01.420000|Number of failed test exceeds the threshold|test
gz-msgs11-debbuilder|94|2024-01-17 09:47:06.325000|Error in configuration dependencies|create deb packages
gz_common-main-win|15|2024-01-16 22:06:24.088000|Jenkins channel closing down (Agent disconnected)|colcon compilation with tests for gz-common6
gz_gui-main-win|17|2024-01-16 22:28:36.389000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
gz_gui-main-win|16|2024-01-16 22:06:24.141000|Jenkins channel closing down (Agent disconnected)|colcon compilation without test for dependencies of gz-gui9
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in gz_math-7-win
- Failing 25.0% in 1 out of 4 builds
	Stopwatch.StartStopReset

Issue #2 in gz_sim-7-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_multiple_servers.test_ran
	  (also in: "gz_sim-ign-gazebo6-win" job)

Issue #3 in nightly_linux_debug
- Failing 6.25% in 1 out of 16 builds
	ros2topic.ros2topic.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #4 in nightly_linux_repeated
- Failing 6.67% in 1 out of 15 builds
	test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[Strings] (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	projectroot.test_publisher_subscriber__rclcpp__rmw_fastrtps_cpp__rmw_cyclonedds_cpp

Issue #5 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_services_cpp
	projectroot.test_services_cpp__rmw_fastrtps_cpp

@github-actions GitHub Actions
Copy link
Contributor Author

Jan 18

Build regressions 
packaging_linux|3302|2024-01-18 08:31:06.676000|Early fail|No last section
packaging_linux|3301|2024-01-18 07:57:03.337000|Jenkins channel closing down (Agent disconnected)|build underlay packages
nightly_linux-rhel_debug|1790|2024-01-18 04:00:01.114000|Jenkins channel closing down (Agent disconnected)|build
Hci__nightly-connext_ubuntu_jammy_amd64|203|2024-01-18 07:15:00.945000|Jenkins channel closing down (Agent disconnected)|build workspace in isolation
gz-msgs11-debbuilder|95|2024-01-18 09:56:04.724000|Error in configuration dependencies|create deb packages
gz_rendering-7-win|14|2024-01-17 16:19:19.834000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
gz_rendering-7-win|13|2024-01-17 16:06:23.690000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
gz_sim-7-win|20|2024-01-17 17:19:38.722000|Jenkins channel closing down (Agent disconnected)|configure the MSVC compiler
gz_sim-7-win|19|2024-01-17 17:06:09.406000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
gz_rendering-ign-rendering6-win|13|2024-01-17 21:59:11.084000|Unknown Reason|clean up workspace
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in gz_math-7-win
- Failing 25.0% in 1 out of 4 builds
	Stopwatch.StartStopReset

Issue #2 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	projectroot.src.core.ddsc.tests.CUnit_ddsc_userdata_participant

Issue #3 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	rosidl_typesupport_introspection_tests.ArraysMessageIntrospectionTest/0.CanWriteTypeErasedMessage
	projectroot.test_arrays_message_introspection

@github-actions GitHub Actions
Copy link
Contributor Author

Jan 19

Build regressions 
nightly_linux-aarch64_debug|2647|2024-01-19 05:00:20.152000|Early fail|No last section
nightly_linux-aarch64_debug|2646|2024-01-19 04:00:01.132000|Jenkins channel closing down (Agent disconnected)|build
nightly_win_deb|2977|2024-01-19 04:00:01.601000|Number of failed test exceeds the threshold|test
gz-msgs11-debbuilder|96|2024-01-19 09:47:59.106000|Error in configuration dependencies|create deb packages
gz_common-5-win|14|2024-01-19 02:06:22.882000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in gz_sim-main-win
- Failing 33.33% in 2 out of 6 builds
	INTEGRATION_mesh_inertia_calculation.test_ran

Issue #2 in gz_math-7-win
- Failing 33.33% in 1 out of 3 builds
	Stopwatch.StartStopReset

Issue #3 in gz_sim-ci-main-jammy-amd64
- Failing 16.67% in 1 out of 6 builds
	UNIT_Gui_TEST.test_ran

Issue #4 in nightly_linux_debug
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_play_services__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/rosbag2/issues/862 - CLOSED)
	  (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rosbag2/issues/862 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/rosbag2/issues/862 - CLOSED)

Issue #5 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	rosidl_typesupport_introspection_tests.ArraysMessageIntrospectionTest/0.CanWriteTypeErasedMessage
	projectroot.test_arrays_message_introspection

@github-actions GitHub Actions
Copy link
Contributor Author

Jan 20

Build regressions 
nightly_win_deb|2978|2024-01-20 04:00:01.647000|Number of failed test exceeds the threshold|test
Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64|89|2024-01-19 23:05:29.967000|Jenkins channel closing down (Agent disconnected)|build workspace in isolation and install
Rci__nightly-release_ubuntu_jammy_amd64|718|2024-01-19 18:05:14.687000|Number of failed test exceeds the threshold|run tests
gz-msgs11-debbuilder|97|2024-01-20 09:47:26.439000|Error in configuration dependencies|create deb packages
gz_launch-ci-gz-launch7-homebrew-amd64|14|2024-01-19 12:06:09.966000|Early fail|No last section
gz_sensors-ci-gz-sensors8-jammy-amd64|40|2024-01-19 12:14:00.510000|Jenkins channel closing down (Agent disconnected)|see Dockerfile
sdformat-sdf14-win|13|2024-01-19 14:06:19.092000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 13.33% in 2 out of 15 builds
	test_communication.TestRequesterReplierAfterShutdown.test_processes_finished_gracefully[BasicTypes]
	projectroot.test_requester_replier__rclcpp__rmw_fastrtps_cpp

Issue #2 in nightly_win_rep
- Failing 13.33% in 2 out of 15 builds
	projectroot.test_publisher_subscriber__rclpy__rmw_fastrtps_cpp (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)

Issue #3 in nightly_win_rep
- Failing 13.33% in 2 out of 15 builds
	rclcpp.TestTimersManager.timers_thread
	projectroot.test.rclcpp.test_timers_manager

Issue #4 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_Server_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2282 - OPEN)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2282 - OPEN)

Issue #5 in gz_sim-ci-main-jammy-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_Gui_TEST.test_ran

Issue #6 in gz_sim-main-win
- Failing 20.0% in 1 out of 5 builds
	INTEGRATION_mesh_inertia_calculation.test_ran

Issue #7 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.TestTwoExecutablesAfterShutdown.test_node_name
	projectroot.test_node_name__rmw_fastrtps_cpp

Issue #8 in nightly_win_rep
- Failing 6.67% in 1 out of 15 builds
	test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[MultiNested] (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED)
	test_communication.TestPublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time[MultiNested]

@github-actions GitHub Actions
Copy link
Contributor Author

Jan 21

Build regressions 
gz-msgs11-debbuilder|98|2024-01-21 09:49:15.502000|Error in configuration dependencies|create deb packages
gz_rendering-ci-main-jammy-amd64|23|2024-01-20 17:06:04.878000|Unknown Reason|compiling
Test regressions 
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 33.33% in 5 out of 15 builds
	rclcpp.TestGuardCondition.construction_and_destruction
	projectroot.test.rclcpp.test_guard_condition

Issue #2 in nightly_win_rep
- Failing 20.0% in 3 out of 15 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp

Issue #3 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_minimal_scene.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/562 - CLOSED)
	INTEGRATION_marker_manager.test_ran

Issue #4 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_Server_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2282 - OPEN)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2282 - OPEN)

Issue #5 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 16.67% in 1 out of 6 builds
	ServerRepeat/OdometryPublisherTest.GaussianNoiseTest/0

Issue #6 in gz_sim-ci-main-jammy-amd64
- Failing 20.0% in 1 out of 5 builds
	UNIT_Gui_TEST.test_ran

Issue #7 in gz_sim-main-win
- Failing 20.0% in 1 out of 5 builds
	INTEGRATION_mesh_inertia_calculation.test_ran

@github-actions github-actions bot mentioned this issue Jan 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
buildfarmer-log Log of issues in the buildfarm
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Crola1702