ROS 2 executors and callback groups explained

Why an executor exists at all

A ROS 2 node on its own does nothing. It registers entities: subscriptions, timers, service servers, action servers. Something has to watch those entities and invoke the right user function when a message arrives or a timer expires, and that something is the executor. The concepts page states it directly: an executor uses one or more threads of the operating system to invoke the callbacks of subscriptions, timers, service servers, action servers and similar entities (Executors, Jazzy docs).

If you are new to the framework, What is ROS 2? covers the node and middleware layers, and ROS 2 topics vs services vs actions covers the interfaces whose callbacks are being scheduled here.

One design decision explains most of the behaviour downstream. Unlike ROS 1, an incoming message is not copied into a queue inside the client library. It stays in the middleware until a callback takes it, and a wait set with one binary flag per queue tells the executor that data is available. Timers are detected through the same mechanism. Since the executor learns only that a topic has messages waiting, not how many or in which order, its scheduling rules are looser than a plain queue would suggest. The depth of those middleware queues is a Quality of Service decision, covered in ROS 2 Quality of Service policies explained.

The one line main functions hide

The simplest C++ program ends with rclcpp::spin(node), which the documentation expands into an explicit executor:

rclcpp::executors::SingleThreadedExecutor executor;
executor.add_node(node);
executor.spin();

The Python equivalent, rclpy.spin(node), looks like the same idea and is, with one detail worth knowing. When no executor is passed, rclpy.spin() and rclpy.spin_once() use a process wide global executor, and that global executor is created as a SingleThreadedExecutor and shut down with the context (rclpy __init__.py, rolling). So the familiar one line Python main loop is single threaded by construction, and its concurrency story only changes when you stop using it and drive an executor yourself:

executor = MultiThreadedExecutor()
executor.add_node(node)
executor.spin()

That pattern comes from the documentation’s own callback group example (Using callback groups, Jazzy docs). The executor also exposes spin_once(timeout_sec=...) and spin_until_future_complete(...), which is how a program keeps doing other work while the executor drains callbacks.

Three executors in C++, two in Python

rclcpp provides three executor types derived from a shared base class:

• SingleThreadedExecutor, the default behind spin(node) and also the executor used by the container process that runs components.
• MultiThreadedExecutor, which creates a configurable number of threads so several messages or events can be processed in parallel.
• StaticSingleThreadedExecutor, which scans a node’s structure (subscriptions, timers, service servers and so on) once when the node is added instead of regularly, cutting runtime overhead. The documentation limits it to nodes that create all their entities during initialization.

Any of the three can serve several nodes: call add_node for each and one thread or thread pool drives them together. How much of that pool a single node actually uses depends entirely on its callback groups.

rclpy does not mirror the static variant. The module defines the Executor base class plus SingleThreadedExecutor and MultiThreadedExecutor (rclpy executors.py, rolling). The multi threaded constructor takes num_threads; when it is left as None, rclpy asks the operating system how many CPUs the process may use and falls back to os.cpu_count(), or to 2 if neither answer is available. Ask for a single thread and it warns, suggesting the single threaded executor instead. There is a matching constraint on the other side: the spin_once docstring notes it should not be called from multiple threads with the same node or executor argument.

Callback groups

ROS 2 organises a node’s callbacks into groups, and there are two types, chosen at construction time:

• Mutually exclusive: callbacks of the group must not be executed in parallel. In rclpy this is implemented with a lock around a single active entity (rclpy callback_groups.py, rolling).
• Reentrant: callbacks of the group may be executed in parallel, including several instances of the same callback.

Callbacks belonging to different groups, of any type, may always run in parallel.

Creating a group differs between the client libraries. In C++ the node owns the factory, create_callback_group(rclcpp::CallbackGroupType::MutuallyExclusive). In Python you construct the group object, MutuallyExclusiveCallbackGroup() or ReentrantCallbackGroup(), and pass it when creating the entity:

my_callback_group = MutuallyExclusiveCallbackGroup()
my_subscription = self.create_subscription(
    Int32, "/topic", self.callback, qos_profile=1, callback_group=my_callback_group)

The callback_group keyword exists on create_subscription, create_timer, create_client and create_service, and each of them falls back to the node’s default group when the argument is omitted (rclpy node.py, rolling). That default is created as a MutuallyExclusiveCallbackGroup, is exposed as node.default_callback_group in Python and as NodeBaseInterface::get_default_callback_group() in C++, and it is where every entity you did not explicitly classify ends up. The documentation also warns that you must keep a reference to a group you created, otherwise the executor will not trigger its callbacks.

One subtlety: an entity relays its callback group to every callback it spawns. The documentation gives the action client as the example, where all callbacks created by the client carry the group assigned to the client.

The rule that catches most people out

Put those two halves together and the practical consequence follows: if everything in a node uses the same mutually exclusive group, the node behaves as if a single threaded executor handled it, even when a multi threaded executor is spinning it. That sentence is in the documentation, and it explains the common complaint that a MultiThreadedExecutor changed nothing. The executor’s threads exist, but the default group serialises every callback they could have taken. Assigning groups deliberately is what makes the executor choice mean anything.

Deadlocks from synchronous calls

The classic failure is a synchronous service or action call made inside a callback. The documentation states the rule plainly: the calling callback and the client making the call must belong to different callback groups of any type, or both must be in a reentrant group, and failing that always causes a deadlock.

The mechanism is worth spelling out, because the blocking code is often invisible. In rclpy, a synchronous Client.call(request) registers a future done callback that the executor has to run before the call returns. If the timer callback and the service client share the node’s mutually exclusive group, the timer callback is holding the group while it blocks, so the done callback that would complete the call can never be scheduled. The ready made fixed version of the documentation’s demo gives the client and the timer separate groups, and the docs note the C++ service client is different in shape: rclcpp has no synchronous call method like the Python one, so C++ code waits on the future returned by async_send_request, with the same group constraint.

Execution order is not FIFO

Another documented behaviour that shapes debugging sessions: while callbacks finish faster than messages arrive, the executor processes work essentially in FIFO order. Once callbacks take longer than the arrival period, the backlog sits in the middleware, and because the wait set reports only whether a topic has messages, the executor visits ready topics round robin, not in the order the samples arrived. The documentation points to the analysis by Casini and colleagues at ECRTS 2019 for the formal treatment, and notes that timer events used to be prioritised over other messages, a behaviour removed in Eloquent.

When an executor is the wrong tool

The documentation is candid about the limits, listing four issues for real time use: complex and mixed scheduling semantics that resist formal timing analysis, priority inversion where a higher priority callback waits on a lower priority one, no explicit control over execution order, and no built in trigger control per topic. Executor overhead in CPU and memory is called considerable. Two alternatives are named. The rclcpp WaitSet lets a program wait directly on subscriptions, timers and service servers instead of handing control to an executor, which allows deterministic processing sequences; the examples_rclcpp_wait_set package demonstrates it. The rclc executor from the C client library, developed for micro-ROS, gives fine grained control over callback order and custom trigger conditions. For the narrower goal of prioritising one control loop over other callbacks, the examples_rclcpp_cbg_executor package shows callback groups distributed across separate executors, which is the intended use of add_callback_group on the C++ executor.

Pitfalls to check before blaming the middleware

• A multi threaded executor with no callback groups. Everything stays in the default mutually exclusive group and the node runs serially.
• A dropped reference to a callback group. The executor will not run callbacks whose group is no longer held.
• Blocking inside a callback. A sleep, a synchronous remote call or a slow computation holds its group, and with a mutually exclusive group that stalls every other callback sharing it. Long callbacks also let middleware queues grow, and QoS depth decides when samples start being dropped.
• Synchronous waits in service or action clients inside callbacks. Different groups or a reentrant group, otherwise deadlock by design.
• Expecting arrival order under load. Above the arrival rate the executor goes round robin between ready topics.
• Calling spin_once from several threads on the same node or executor. The rclpy docstring rules that out.

Sources

• docs.ros.org: Executors (Jazzy)
• docs.ros.org: Using callback groups (Jazzy)
• rclpy executors.py (rolling)
• rclpy callback_groups.py (rolling)
• rclpy node.py (rolling)
• rclpy __init__.py (rolling)
• rclcpp single_threaded_executor.hpp (rolling)
• rclcpp wait_set.hpp (rolling)
• ros2/examples: cbg_executor (rolling)
• Casini et al., response time analysis of ROS 2 processing chains, ECRTS 2019