ROS 2 Executor Models and Callback Scheduling

ROS 2 software is organized as nodes, and each node holds callbacks that fire when a message arrives, a timer elapses, or some other event triggers them. The executor is the piece of software that sits between those callbacks and the operating system: it figures out which callbacks are ready to run, holds them in some order, and hands them off to OS threads for execution. Because every single callback in a ROS 2 program passes through this one mechanism, how the executor makes its decisions ends up deciding, more than any other part of the stack, whether a robot meets its deadlines. Research surveying real-time support in ROS 2 treats the executor as the central object worth studying for exactly this reason: it is the one place where application logic and the operating system's scheduling guarantees actually meet.

Callbacks are also grouped into callback groups, which give the executor a coarser unit to reason about than individual callbacks, and which is how ROS 2 lets a developer say "these things may run concurrently" or "these things must not." Everything that follows, the two-phase loop, the thread assignment schemes, the wait set, the alternative designs, is really an argument about whether that lever is fine enough, and whether the executor built around it can hold up under real timing pressure.

The Two-Phase Scheduling Loop

The default ROS 2 executor decides what to run next through a loop with two distinct phases, and the shape of that loop is where the system's implicit priority rules and its refusal to preempt both come from. The first phase is a polling point: the executor asks a wait set which callbacks, across every timer, subscription, service, and client it manages, currently have something ready to process. The second phase is the processing window: the executor works through whatever came back ready, and once it starts a given callback, that callback runs to completion with no interruption. There is no preemption anywhere in this design, so a callback that takes a long time holds the loop hostage for its entire duration before the executor can poll again.

Picture two subscriptions and a timer all becoming ready at nearly the same instant. The executor's polling point catches all three in the same batch, because the wait set reports readiness as a snapshot, not as a stream of individual arrivals. Inside the processing window that follows, timer callbacks are dispatched ahead of subscription callbacks, and subscription callbacks ahead of service, client, and waitable callbacks, so the class of a callback decides its rank before anything about urgency does. A callback's turn comes from whether it happened to be available in the current batch, not from how urgently it needs to run, producing a de facto round-robin policy across classes. Research surveying ROS 2's real-time properties notes that the scheduling algorithm had to be reverse-engineered from the executor's source code, since the official documentation never specified it, a gap that held back formal timing analysis of ROS 2 systems for years.

What the SingleThreadedExecutor and MultiThreadedExecutor each sacrifice

The two standard executors, SingleThreadedExecutor and MultiThreadedExecutor, solve the same problem, how to map callbacks onto OS threads, in opposite ways, and each choice gives up something a real-time system needs. SingleThreadedExecutor puts every callback on one OS thread. This cost appears directly in execution order. Callbacks run one after another with no preemption, so a long-running callback that happens to carry low real-world priority can block every high-priority callback waiting behind it in the queue, for the full duration of its run.

MultiThreadedExecutor takes the opposite approach and spreads work across a pool of OS threads, but it assigns each callback invocation to a thread from that pool at random. MultiThreadedExecutor arrives at the same architectural problem as SingleThreadedExecutor from the other direction, since per-callback OS-level scheduling control is impossible either way. An intermediate design, the StaticSingleThreadedExecutor, existed in ROS 2 for a time as a variation on the single-threaded model, carrying similar limits around per-callback control. That executor has since been removed from current distributions, a change covered later alongside the alternatives that followed it.

The Wait Set Bottleneck

A second, independent source of unpredictability sits inside the wait set itself, layered on top of whatever the thread-callback mapping has already cost the system. The wait set only gets updated once it becomes completely empty. A callback that becomes ready while the executor is still working through the current batch has to wait, not just for its turn, but for the entire current batch to finish before the wait set will even register that it exists. That is a structural delay, built into when the executor is willing to look again, not an accident of bad luck on a particular run. On top of that, the wait set is protected by a lock, and that lock introduces its own blocking and context-switching overhead whenever more than one part of the system tries to touch it at the same time.

Put those two mechanisms together: a high-priority callback that arrives an instant after a processing window begins can be forced to wait out the full execution time of every single callback in that window before the executor acknowledges it exists, and nothing in the executor's design puts a ceiling on how long that wait can run. That is priority inversion with no upper bound: it is a different problem from the implicit class-based ordering described earlier, because that ordering decides who goes first once everyone is visible to the scheduler, while the wait set decides when anyone becomes visible to it. Research on ROS 2 scheduling, including work attributed to Liu et al., documents and characterizes the wait set lock directly as a bottleneck, not a theoretical concern raised in the abstract.

Priority Inversion in Autonomous Driving Stacks

Autonomous driving software is close to a worst-case workload for the default executor, and that is not a coincidence, since the workload that stresses the executor hardest is exactly the one a driving stack is built from: many concurrent processing chains, running at different levels of criticality, all converging on deadlines measured in milliseconds. The round-robin policy described earlier has no built-in notion that a control callback matters more than a logging callback, so a low-criticality callback from one chain can sit in the way of a high-criticality callback from another, purely because of which one happened to be ready first.

Research on the LaME executor, presented at RTNS 2025 by Enright, Sobhani, and Kim, describes this failure mode in direct terms: without any explicit notion of priority, the time it takes a chain to respond stretches out as the system's overall workload grows, and the obvious fix, just assigning priorities, carries its own cost. Giving priority to some chains can starve the best-effort chains that are left without it. Neither outcome, unbounded growth in response time or outright starvation of lower-priority work, is something a safety-critical system can tolerate. Autonomous vehicles and safety-critical robotics, the deployments most exposed to this failure mode, are also the fastest-growing part of the ROS 2 ecosystem, making the executor's scheduling behavior a live engineering problem today.

The design space of alternative executors and the core question they each answer differently

Every alternative executor proposed in the research community is answering a version of the same question: which layer of the system should be trusted to make scheduling decisions, and how fine-grained should its control be? That question splits along two axes. The first is where the decision gets made, inside ROS 2's own middleware, in user space, or down in the operating system kernel. The second is the unit the decision applies to: a whole chain of callbacks, a single callback, or an entire DAG of callbacks spanning multiple chains.

One school of thought holds that real scheduling guarantees, actual preemption and per-callback control, can only come from the OS kernel, since user-space code is fundamentally limited in what it can tell the OS to do. The other school holds that the kernel is unnecessary if the user-space scheduling algorithm itself is designed correctly, and that keeping scheduling logic in user space has a real advantage: it preserves compatibility with the standard ROS 2 API, so existing code keeps working without modification. A separate, more practical line divides the field again. Others are built to work as drop-in replacements for the standard executors, with no changes required to existing nodes. That distinction, configuration burden versus drop-in compatibility, matters as much to a team deciding what to adopt as whether scheduling decisions are made in the kernel or in user space.

Alternatives that keep scheduling in user space: rclc, LaME, ReDAG-RT, and the events queue approach

The rclc Executor, built for micro-ROS, gives developers a fixed execution order and deterministic scheduling, and it can reach down and configure the OS scheduler directly to set custom task priorities. That capability comes with a condition: it requires explicit configuration from the developer for each callback, so it does not work as a transparent replacement for executors running unmodified ROS components.

The EventsExecutor, added to the ROS 2 distribution in 2023, replaces the wait set entirely with a FIFO queue of events. The EventsCBGExecutor, introduced in the ROS 2 Lyrical Luth distribution, extends that same events-queue idea with awareness of callback groups. Lyrical Luth also marks the point where the StaticSingleThreadedExecutor disappears from the distribution entirely: the rclcpp changelog confirms it was removed in version 29.6.1, and the deprecation notice in the source pointed developers toward SingleThreadedExecutor as the direct replacement, even as EventsCBGExecutor arrived separately as a new option in the same release.

LaME, the Latency Management Executor presented at RTNS 2025 by Enright, Sobhani, and Kim of UC Riverside, takes a different approach built around what it calls thread-classes. Thread budgets and the allocation of chains to threads adjust dynamically while the system runs, with LaME continuously monitoring response times and adapting as workload surges or the CPU throttles under thermal pressure. Tested on the Autoware reference autonomous driving stack running on an Nvidia Jetson platform, LaME manages to hold onto starvation freedom for best-effort chains while still giving real-time guarantees to priority chains, directly answering the starvation problem that simple priority-based fixes create.

ReDAG-RT extends the same user-space philosophy to systems with multiple concurrent DAGs. It introduces a global ready-queue executor that enforces deterministic arbitration across callbacks from different DAGs, requires no modification to the kernel, and preserves full compatibility with the standard ROS 2 API. It was evaluated against both SingleThreadedExecutor and MultiThreadedExecutor in ROS 2 Humble using synthetic multi-DAG workloads, and it adds asymmetric concurrency bounds for each DAG to cut down on the interference one DAG's callbacks can cause another's. What ties all four of these designs together is the events-queue insight the 2023 EventsExecutor formalized: once callback readiness is tracked as a stream of discrete events rather than a polled batch, a large body of classical real-time scheduling theory, developed long before ROS 2 existed, becomes directly usable, and worst-case timing bounds that were previously out of reach become derivable with only minor adaptation.

Alternatives that push scheduling to the OS kernel: PiCAS, CIE, and ROS^RT

PiCAS is the paper that started this entire line of research, by identifying precisely how the standard executor's scheduling algorithm can threaten the schedulability of time-critical callbacks. Its proposal combined fixed-priority scheduling inside a single-threaded executor with a chain-aware node allocation algorithm designed to cut down on interference between competing chains. PiCAS set off a long run of subsequent executor proposals that followed its lead, though it shares the same practical limitation as rclc: it requires explicit user configuration, which keeps it out of reach for unmodified ROS components.

The Callback Isolated Executor, or CIE, developed by TIER IV for Autoware and presented at IEEE RTAS 2025, takes the kernel-side argument to its logical endpoint. Every callback gets a persistent, one-to-one mapping to its own dedicated OS thread, which removes the middleware scheduling layer from the picture entirely and lets the OS apply scheduling policy, priority, and CPU affinity to each callback individually, the exact control that SingleThreadedExecutor and MultiThreadedExecutor were both architecturally unable to offer. A cost analysis by Ishikawa-Aso et al. found that CIE's overhead stays lower than MultiThreadedExecutor's across nearly every metric tested, regardless of how many callbacks are involved, with one narrow exception around user-kernel mode switches under intra-process communication. Measured against SingleThreadedExecutor, the overhead cost holds to a fixed ratio of 1.4x for inter-process communication, and to a larger but still bounded ratio for intra-process communication. CIE was validated on a robotaxi operating on public roads, where the worst-case response time across five critical paths improved by roughly a factor of five compared to running under Linux's default CFS scheduler with no tuning applied. CIE is designed to ship alongside Agnocast, a zero-copy IPC middleware for ROS 2 that supports every message type, including PointCloud2 data. The two can be adopted independently, but they are built to work together; migration of that pairing across the full Autoware codebase is still in progress.

What changes the calculus for kernel-side designs going forward is the arrival of SCHED_EXT, a Linux kernel feature that lets custom scheduling policies be loaded without writing a kernel module from scratch. Before SCHED_EXT, pushing scheduling authority into the kernel meant accepting real configuration and portability costs, exactly the kind of tradeoff PiCAS and the early kernel-side designs had to make. SCHED_EXT narrows that cost considerably. Kernel-side scheduling for ROS 2 is shifting from a research niche into a design option that ordinary deployments can realistically consider.