ExplorRobo
robot-safetyfunctional-safetyrobot-securityros2vlamobile-robotsregulation

The Stop Function Is Certified. Everything Feeding It Is Not.

By ExplorRobo Editorial · · 11 min read

Figures attributed to companies, projects and papers are their published claims. The synthesis and judgements below are this author’s analysis. I read the incident report, the vendor material and the paper abstracts; I have not run any of these systems.

On the evening of 26 October a delivery robot entered a pedestrian crossing against a red light in central Tallinn. The bus driver braked hard, and the city’s transport company said up to nine standing passengers were injured. Starship Technologies, which builds the robots, gave the cause two days later: an illuminated advertisement stood behind the traffic lights, pedestrian and vehicle signals sat in the same line of sight, and the robot misread which light was lit. Its head of operations added that the robot’s light sensitivity “was not working as it should have and had changed for some reason”.

No motor failed. No message was dropped. No attacker was involved. The obstacle detection had nothing to report, because the crossing was empty and the bus came from the side. The stop function was correct, and the input was wrong in a way that looked correct.

That is the seam this week’s material keeps exposing. A safety case certifies a reaction to a measurement, and the measurement now comes from learned perception, travels over middleware most deployments do not authenticate, and is validated in simulators built from third party assets. Three publications attacked those layers, and one deadline makes the paperwork urgent: the EU Machinery Regulation applies from 20 January 2027, 79 days from today.

A stop zone cannot see a red light

The intuitive reading of Tallinn, that the robot failed to see the bus, is wrong. It saw the road, the crossing and the absence of traffic in the lane it was entering. What it got wrong was a semantic state: which signal was illuminated for whom. A stop zone, a bumper, a time-to-collision estimate and a protective field all ask whether something is in the way, and nothing was.

Starship’s explanation is less comfortable than a miss. The company says the robot’s light sensitivity had changed, which is calibration drift: the sensor kept producing values, and the pipeline accepted them. A vision system that returns garbage usually trips a health check. One whose exposure or colour response has shifted returns a consistent, confident, wrong reading, and the failure surfaces only when that reading maps onto a physical decision.

The decision was made on board, by a robot that believed the light was green, with nothing between the camera and the wheels able to disagree. The back office moved it off the road afterwards. What an incident review needs is whether the advertisement was still recoverable from the logs.

The same shape, with an attacker instead of an advert

The adversarial version of this failure has been measured. FreezeVLA, posted in September 2025, formalises an attack in which an adversarial image “freezes” a vision-language-action model so that it ignores later instructions. Across three state-of-the-art VLA models and four robotic benchmarks the authors report an average attack success rate of 76.2 percent, and the images transfer: one image reliably induced paralysis across different prompts.

It is a lab result on benchmarks, not a field demonstration. It belongs next to a misplaced traffic light because of the structure: an input that is wrong in a way the policy cannot represent as wrong, followed by confident output. The attack does not make the robot do something strange. It makes it stop responding while everything upstream looks healthy.

VicOne’s lab describes the same seam from the other side: a robot dog that treated poster text as an instruction, a hospital service robot in simulation whose behaviour changed under audio people cannot hear, and a bug bounty event where a ROS 2/DDS message injection moved a robot that organisers expected to stay still. That last case does not touch perception. It targets the transport layer between the component that decides and the component that moves.

These are demonstrations by a company selling robot security tooling, published as sponsored content. They establish a method and a plausible failure path, not prevalence.

The evidence pipeline is the soft target

The consequential claim from the same lab is about where weaknesses sit. The Robot Report carried the summary on 1 November: 28 zero-day findings in less than two days of AI-assisted research across software supporting the Physical AI ecosystem, including projects maintained by Google, NVIDIA and Meta, with 75 percent of the findings in development, training, simulation or validation tools rather than in deployed robots.

Treat the count as marketing. The vendor says all 28 are classified as zero days on initial validation, that the count may change under review, and that some findings do not yet have CVE identifiers. A number with no advisory list behind it is a lead, not evidence. The distribution is the argument. A weakness in a simulator attacks the safety case at its source, because that case is a claim about evidence: this policy was tested on these scenarios and passed. VicOne describes a compromised scene giving an attacker control over simulated sensor readings and results, so that a robot appears to pass tests that should have exposed unsafe behaviour. The conformity file would then be a true statement about false data.

The same reports show why the remote stop is not the refuge it looks like. VicOne reports exploiting a vulnerability to prevent a robot’s remote-stop command from working. That command is the function everyone treats as out of band, independent of the autonomy stack. If it runs over the same network and the same supply chain, it is not out of band, only less frequently exercised.

Safety filters certify less than the word suggests

The research answer to learned policies is the safety filter, and the week’s most candid paper on it names the limit. Confidence-Modulated Safety Filtering for End-to-End Policies, by Javier Borquez and Jaime Fisac, splits the system into two domains: the filter’s operational design domain, where its assumptions hold and safety is certified, and the policy’s competence domain, where the policy is reliable. The two do not coincide, and a filter with fixed conservativeness trades performance inside the first against margin outside it.

Their proposal lets the policy’s confidence set that trade at runtime: confidence buys speed inside the design domain, and low confidence adds margin beyond it, which helps where confidence actually flags the filter’s model failing. The evidence is preliminary, in F1TENTH simulations, and the authors say so. What travels beyond the simulation is the framing. A certified filter is a certified model of the world, and this world contains a road, a light and a rule that an advertisement has no obligation to respect. When the Tallinn robot decided the light was green, no competence estimate was consulted and no second measurement could outvote the camera.

The deadline is 20 January 2027

The European Commission’s machinery page states the change plainly. Regulation (EU) 2023/1230 was adopted on 14 June 2023 and applies on a mandatory basis from 20 January 2027. Among its changes, in the Commission’s summary, it “integrates provisions for machinery with safety functions that are AI-powered” and “integrates provisions for cyber-safety for compliance-relevant software data and safety control systems”. Machinery placed on the market before that date stays under Directive 2006/42/EC.

Whatever the harmonised standards eventually specify, the direction is explicit: the technical file has to say something about software and data, and about a safety function whose inputs are corrupted. The current file describes stopping distance, detection field, reaction time and controller fault behaviour. It rarely describes which model version was loaded, which simulator validated it, what perception does when calibration drifts, or how the stop command is authenticated. The audit question that finds real problems is whether one action can mislead several inputs at once, because guards that share a network, a clock or a model are not redundant.

What this means if you are building

  • Build a second sensing path, not a second opinion. Nav2’s Collision Monitor has the shape: its README says it bypasses the costmap and trajectory planners, filters the cmd_vel leaving the controller server, and works from laser scans, point clouds or range sensors. It also says it “does not provide hard real-time safety certifications”, so use it as a differently sensed channel rather than a certificate. See how the Nav2 stack fits together.
  • Make the guard depend on physics that perception cannot restyle. Contact bumpers, a downward range sensor, wheel odometry residuals, a geofence. Each is boring, and each is harder to make confidently wrong than a camera.
  • Turn on SROS2 if you have not, and pin what you load. The sros2 repository describes the package as the tools to run ROS 2 on top of DDS-Security. It is a deployment step with a keystore and enclaves, so unless a team performs it the transport layer has no authentication to offer. Hash the checkpoints and simulator assets you accept, record their identifiers next to the test result, and bag the perception verdict and guard state beside the raw topics, or incident review cannot separate a lying sensor from an erring policy. Background in What is ROS 2?.
  • Put the guard below the planner. The guard layer still has no home argued that refusal belongs in the executable action; the regulation gives that layer a filing cabinet, not a sensor. Veloxity, a Rust autopilot for UAVs posted on 29 October, points at the longer term fix underneath, with reported parity against C++ equivalents in simulation, testbenches and outdoor flights.

What to watch

Whether the 28 findings become advisories naming affected versions, the first public map of the Physical AI toolchain’s weak points. Whether Starship says what changed in its light handling, calibration or an added check. Whether an early conformity assessment publishes adversarial test evidence rather than a summary of a passing test. And whether confidence signals survive a wet street at dusk, where the filter literature has to prove itself next.

Sources

Frequently asked questions

Does a safety certified mobile robot protect against manipulated sensor input?
Not by itself. A stop function is specified as a reaction to a measurement: detect an obstacle inside a field, decelerate inside a distance. If the measurement is wrong but plausible, the reaction is correct and useless. VicOne's robotics lab reports that injecting a ROS 2/DDS message moved a robot that organisers expected to stay still under a safe-control setting.
Is ROS 2 communication encrypted by default?
No. Secured communication needs the sros2 tools plus a keystore and security enclaves, and that is a deployment step. The sros2 repository describes the package as the tools and instructions to run ROS 2 on top of DDS-Security rather than something the middleware does on its own.

Related reading

The Robot Stack Is Going Modular, and the Guard Layer Has No Home

HomeBody deletes the learned policy, FLUX 3 Action shrinks it to a checkpoint, and RoboHarm shows the frontier model that takes over refuses almost nothing.

Atlas Metaplant opens, Halos safety stack

Boston Dynamics starts training Atlas inside Hyundai's Metaplant America, NVIDIA argues safety has to be a whole stack, and robot policies rarely refuse.

Gazebo armhf EOL, NSF physical AI

Gazebo ends 32-bit ARM binaries, NSF opens a physical AI funding topic, Hirebotics adds line tracking and a rail, and a defense for backdoored VLA policies.