NetworkException (UDP receive timeout) thrown from readOnce() during controller switch, causing process crash

[IDavGal]

Summary

During ROS2 controller activation, specifically when FrankaHardwareInterface calls startTorqueControl() inside perform_command_mode_switch(), a franka::NetworkException is thrown with message libfranka: UDP receive: Timeout, which crashes the control loop.

The failure happens sporadically, although sometimes with such a frequency that makes it difficult to work with the robot.

---

Environment

Component	Version
libfranka	0.20.5
franka_ros2	v3.3.0
ros2_control	4.44.0
ROS 2	Jazzy
Robot	FR3
OS	Ubuntu 24.04
Kernel	Real-time kernel

---

For this scenario, we're spawning and activating JointStateBroadcaster and a JointTrajectoryController in the same launchfile. We're using the following controllers.yml configuration file:

controller_manager:
  ros__parameters:
    update_rate: 1500  # Hz
    overruns:
      manage: false
      print_warnings: false
    thread_priority: 98
    handle_exceptions: false

    joint_state_broadcaster:
      type: joint_state_broadcaster/JointStateBroadcaster

    arm_pid_controller:
      type: joint_trajectory_controller/JointTrajectoryController

    franka_robot_state_broadcaster:
      type: franka_robot_state_broadcaster/FrankaRobotStateBroadcaster
      state_interfaces:
        robot_state

franka_robot_state_broadcaster:
  ros__parameters:
    convenience_publish_rate: 1000
    robot_type: "fr3"

arm_pid_controller:
  ros__parameters:
    command_interfaces:
      - effort
    state_interfaces:
      - position
      - velocity
    joints:
      - fr3_joint1
      - fr3_joint2
      - fr3_joint3
      - fr3_joint4
      - fr3_joint5
      - fr3_joint6
      - fr3_joint7
    allow_nonzero_velocity_at_trajectory_end: false

---

Behavior

On an affected run, approximately 1 second after controller activation is requested:

[controller_manager]: Activating controllers: [ arm_pid_controller joint_state_broadcaster ]
[controller_manager]: Exception of type : N6franka16NetworkExceptionE thrown during read
  of the component 'FrankaHardwareInterface': libfranka: UDP receive: Timeout
terminate called after throwing an instance of 'franka::NetworkException'
  what():  libfranka: UDP receive: Timeout
Stack trace (most recent call last) in thread 67448:
#19   Object "", at 0xffffffffffffffff, in
[controller_manager]: Switch controller timed out after 5 seconds!
[controllers_spawner]: Failed to activate the parsed controllers list :
  ['arm_pid_controller', 'joint_state_broadcaster']

The timestamps show:

• T0 -> Activating controllers requested
• T0+1s -> NetworkException: UDP receive: Timeout thrown
• T0+5s -> Switch controller timed out after 5 seconds

---

I understand this is being thrown by the udpBlockingReceive() call made from within the initializeTorqueInterface() inside perform_command_mode_switch() triggered by the controller activation request.

The UDP socket has a 1-second receive timeout set at construction:

// franka_ros2/franka_hardware/src/robot.cpp
robot_ = std::make_unique<franka::Robot>(robot_ip, rt_config);
// → libfranka/src/network.cpp:
udp_socket_.setReceiveTimeout(Poco::Timespan{1000L * udp_timeout.count()});
// udp_timeout defaults to std::chrono::seconds(1)

As a side note, as shown in the controller configuration file, we're setting handle_exceptions: false. This causes exceptions to propagate upstream, and it's intentional in our use case. Regardless of that configuration, the switch_controller request times out and requires a manual restart.

Is this a known race condition that causes the robot stops sending FCI UDP state packets for more than 1 second during startTorqueControl()?
Is there anything in our configuration that could be causing this sporadic behavior?

[AndreasKuhner]

Hi @IDavGal , if you are receiving a libfranka: UDP receive: Timeout, it is likely that network connection is not RT capable. Did you check what ping latency you have from your PC to the PC of the robot? A 'good' connection would have around ~200 us.

Cheers,
Andreas

[IDavGal]

Hi Andreas, thanks for the quick response.
Running the ping test against the robot shows an average RTT of 0.2 ms

---  ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 9999ms
rtt min/avg/max/mdev = 0.119/0.205/0.274/0.028 ms

As mentioned above, the error appears sporadically, with the frequency occasionally spiking. This particular ping test was run between failing runs.

It is also worth noting that recently we upgraded the robot firmware and software packages from versions:
Firmware: 5.7.2 -> 5.9.0
franka_ros2: v3.0.0 -> v3.3.0 |
libfranka: 0.15.0 -> 0.20.5
franka_description: 2.0.0 -> 2.7.0

Before the upgrade this was not (at least noticeable) a problem.

[AndreasKuhner]

Hi @IDavGal ,
0.2 ms looks fine.
Question: Why do you have update_rate: 1500 # Hz?

Yet, for a UDP timeout, something is off with the network. Of course, our RT connection has a high requirement of the latency but having a timeout points to some blocking component. Is maybe another component starving the boot up sequence?

Cheers,
Andreas

[IDavGal]

It is a possibility I'm investigating, although for this issue I made sure to run a minimal setup with only the controllers and hardware interface for the robot being launched.

The update rate was increased after experiencing consistent communication_constraints_violation errors after a short amount of time with the stack running at 1kHz. We set the overruns.manage: false as per what was discussed in #2529, but that didn't seem to fix the occurrences of the violation. Do you think that this could be related to the UDP timeouts? Maybe, as you point out, an issue with the network, or even the hardware?

Thanks!

[AndreasKuhner]

Interesting -> the overrun mechanism would interact with a higher frequency. So, if you guys increased it and it improved the situation, I have a feeling that this overrun mechanism maybe doesn't work anymore as expected...

We need to investigate.