Depth Camera Selection for Robot Manipulation and Navigation

A depth camera decides if a robot arm can find the edge of a bolt hole and if a mobile robot sees the pallet jack backing into its path in time to stop. Get the choice wrong, and the failure appears months later, not on day one. The failure occurs months later, when a wrist camera that can't resolve range past a meter gets asked to do bin picking on a bin deeper than a meter, or when a warehouse robot's navigation stack loses the corridor because its active stereo pattern gets washed out by a skylight.

Depth cameras feed the point clouds and RGB-D frames that grasp planners, SLAM pipelines, obstacle-avoidance stacks, and learned visual policies all depend on. Manipulation and navigation ask that sensor to do two different jobs, and a camera tuned for one job fails at the other on a regular basis, not as an edge case. Research on the SpatialPoint framework, built by Visincept together with Tsinghua and IDEA researchers and trained on a large-scale RGB-D dataset, found that feeding structured depth into vision-language models measurably improves how well a system localizes itself in space. The finding matters past that one project: sensor quality sets a ceiling on what any downstream policy or planner can reach, no matter how good the algorithm underneath it is.

What the three sensing architectures trade away

Every depth camera on the market today builds depth one of three ways: active stereo, time-of-flight, or structured light. Each method turns light into distance through a different physical process, and each one breaks in its own predictable way. None of the three is a safe default. Picking one because it's familiar, rather than because the failure mode matches the deployment, is the single most common mistake in this space.

Active stereo pairs two image sensors with a projected infrared pattern, the same basic geometry as human binocular vision, but with texture painted onto the scene by an IR projector so flat, featureless surfaces still produce a match between the two images. When the pattern can't be decoded, in bright sunlight or at long range, the camera can fall back to passive stereo instead. This is the workhorse architecture for indoor robotics; Intel's RealSense D400 family and Orbbec's Gemini series both build on it. Its failure mode is specular glare: polished floors and glass panels scatter the IR pattern, and the resulting depth holes appear as phantom obstacles inside a navigation costmap.

Time-of-flight cameras (iToF) skip triangulation. They fire IR light and measure how long it takes to bounce back, which gives depth at higher frame rates and longer range than structured light can manage. Accuracy is in the millimeter range generally, a step below what structured light delivers at a short, calibrated distance. The failure mode here is multipath interference: under strong ambient light, reflections off multiple surfaces land on the same pixel and corrupt the reading. iToF is useful in mobile robotics, obstacle avoidance, AMR navigation, and human-machine interaction, in any setting where steady depth across a range of distances is needed over sub-millimeter precision at one fixed point.

Structured light projects a known IR pattern and computes depth by comparing what the camera sees against a calibrated reference image. At short, fixed distances it delivers the highest spatial resolution of the three, a high pixel count of depth data. That precision costs compute and speed: frame rates cap out around 30 fps, and sunlight washes the projected pattern out completely. Stereo vision stays the default choice outdoors as a result. Structured light belongs on fixed-position industrial arms working in controlled light, where the working distance is known ahead of time and precision matters more than speed.

Task requirements that must be defined before evaluating any camera

Before any datasheet gets opened, four questions need answers, and skipping any one of them is how teams end up with a camera that looks great on paper and fails on the floor. Working range comes first: manipulation tasks cluster at centimeter-to-sub-meter distances, while navigation needs reliable coverage from near-field obstacles out to several meters. Accuracy comes next, and it means something different in each context. Manipulation, especially insertion or in-hand pose estimation, needs millimeter-or-finer resolution at the actual working distance. Navigation needs accuracy that holds steady across the whole range envelope rather than peaking at one distance and falling apart everywhere else.

Form factor and mounting come third. A wrist-mounted camera on a dexterous arm has to be small and light enough that it doesn't throw off the arm's dynamics; a camera bolted to a mobile base or an overhead rig has far more room to work with. Environmental tolerance is the fourth axis, and engineers underweight it more often than any of the other three: outdoor or field robots need passive stereo or an active projector with an infrared filter that survives sunlight, and indoor robots working near glass or polished floors need a camera that handles reflection without falling apart.

Secondary constraints producing those four deserve a check early, not after the hardware's already ordered. Some cameras need a CUDA-capable GPU on the host machine; others process depth entirely on board, which matters enormously on embedded or cost-constrained platforms. Frame rate and latency requirements differ by task: reactive manipulation and fast conveyor tracking need higher frame rates than a robot rolling slowly across a warehouse floor. Multi-camera rigs, humanoids, and AMR fleets bring their own connectivity demands (GMSL2, hardware sync, fleet-level camera management), and field robots working outdoors or in agriculture need an enclosure with a real IP rating, not a marketing claim printed on the box.

All of this collapses into one practical split. Manipulation wants close range, high precision, a small body, and usually operates in lighting the engineer controls, which points toward structured light or short-baseline active stereo. Navigation wants longer range, steady performance as distance changes, and tolerance for a scene that keeps moving, which points toward iToF or stereo systems with longer baselines. Everything past this point in the discussion is just working out the implications of that split.

Cameras suited to manipulation: close-range performance and wrist-mount constraints

The RealSense D405 is the reference point for close-range manipulation, and it earns that spot rather than inheriting it. Its ideal operating range runs from 7 cm to 50 cm, with minimum object detection down to 500 microns at the 7 cm mark. It captures depth at a high resolution and frame rate and hits defect-detection accuracy as fine as 0.1 mm. At 42x42x23 mm and around 51 grams, it's the smallest, lightest camera in the RealSense line, built specifically to sit on a wrist without throwing off the arm's balance. It has become a widely referenced choice for wrist-mount RGB-D sensing, applied to close-range tasks where its precision and compact form factor matter most. Intel's RealSense Perception Studio beta pushes the same sensing down to under 3 cm on the D401 and D405 through an SDK update, tightening the close end of the envelope even further.

That precision costs range: usable distance tops out around 1.5 meters, and the field of view is narrower than the D435i's. For pure close-range grasping or insertion, that's a trade worth making without hesitation. It stops being acceptable the moment a task needs both close-range grasp accuracy and awareness of a wider scene: bin picking in a deep bin, or inspecting a large assembly where the camera has to see both the part in the gripper and the fixture around it.

Orbbec's ToF-based offerings answer that gap directly. They give up some of the D405's raw precision in exchange for more range headroom, and Orbbec's lineup is fully supported on a common robotics middleware framework, which matters for anyone trying to swap sensors without redesigning a mount. Structured light also has a place in manipulation, specifically fixed-arm work in controlled environments like semiconductor pick-and-place, where the working distance never changes and the 30 fps ceiling isn't a constraint because the task was never fast to begin with.

Cameras suited to navigation: range, consistency, and environmental resilience

Navigation asks for a different set of trade-offs, and the RealSense D435i has long served as the general-purpose answer for indoor SLAM. It offers an 86 by 57 degree field of view, error under 2% at 2 meters, and an integrated IMU that plugs directly into SLAM pipelines. Its weakness appears the moment it goes outdoors, where the active-stereo IR pattern loses to sunlight. A Czech Technical University study found the D435 performs best at tabletop range but gets outperformed by other cameras at longer distances, which makes it a weak primary sensor for any mobile platform that needs to sense obstacles 3 to 4 meters out.

The RealSense D455 extends that envelope in a way that actually matters for corridor-scale navigation. It offers depth range out to 6 meters, extending its useful envelope for corridor-scale navigation. It's still active stereo underneath, though, so sunlight and reflective surfaces remain the same hazard they were on the D435i.

Stereolabs' ZED 2i targets the outdoor and long-range case directly. If the deployment is outside, it's the stronger starting point over either RealSense option above. It captures up to a high resolution at 15 fps or a lower resolution at 60 fps, with a 110-degree horizontal, 70-degree vertical, 120-degree diagonal field of view and depth range out to 20 meters. It carries its own IMU, barometer, magnetometer, and temperature sensor, runs a neural depth engine, and is built for demanding field conditions, aimed squarely at outdoor and field robotics work. In the same Czech Technical University comparison, the ZED 2 demonstrated strong long-range accuracy, holding error under 3 cm even at 4 meters. It isn't a standalone device, though: it needs a CUDA-capable NVIDIA GPU with Compute Capability 7.5 or higher, which rules out CPU-only platforms outright, and it costs noticeably more than comparable RealSense or Orbbec hardware.

Stereolabs' ZED X pushes further into industrial and field territory, built Jetson-native from the start. It comes in an IP66-rated aluminum enclosure, supports GMSL2 and native multi-camera sync, and captures global shutter RGB at a high resolution at up to 120 fps with a 3.0 micron pixel size tuned for both low light and bright conditions. Its navigation range runs 0.2 to 20 meters, with object detection from 0.08 to 12.5 meters. It connects to an NVIDIA Jetson system, and a single Jetson AGX Orin can drive multiple ZED X units at once, which cuts the per-camera cost, weight, and footprint significantly on any rig running multiple cameras.

The D585 Pro's architecture and what it signals about where the field is heading

Unveiled at Automate 2026, with shipping expected in the first quarter of 2027, the D585 Pro deserves attention less for any single spec than for how directly it targets the trade-offs laid out above. Its minimum range comes in under 15 cm at full resolution, a figure that represents a meaningful improvement over prior RealSense generations, and that number starts closing the gap between close-range manipulation sensing and wider-scene navigation sensing inside one device instead of two. Its field of view runs 120 by 100 degrees at 60 fps, double the frame rate of 30 fps-class competing cameras, and its operating range extends past 10 meters at full accuracy, enough to cover a typical AMR's full navigation envelope on its own.

Every unit ships with IP65 as a standard rating rather than a premium add-on, and it carries built-in IR filters meant to hold up in both bright outdoor sunlight and dim indoor conditions. Dual active projectors are built in specifically to resist harsh ambient light and the repetitive surface patterns that tend to defeat passive stereo. Connectivity covers both GMSL2 and USB-C, with hardware sync support built in from the start.

The bigger shift sits inside the camera itself, not on its spec sheet. A proprietary Gen 5 system-on-chip handles the depth engine, image signal processing, DSP work, and AI acceleration on a quad-core ARM processor, and the manufacturer states it delivers more than double the depth quality of the previous RealSense generation, all without leaning on host compute. Paired with that is a software-defined approach: the Perception Studio SDK is built to push new capabilities (visual-inertial odometry, occupancy grid generation, auto-calibration, people detection) to cameras already in the field after launch. That changes how a camera's useful life gets measured. A sensor bought today doesn't need replacing to gain tomorrow's feature; it just needs a software update, the same logic that reshaped how phones and cars get sold.

Environmental and deployment conditions that override datasheet comparisons

None of the numbers above mean much if the deployment environment breaks the sensor's core assumption. Outdoor sunlight kills structured light. The fix is active stereo with IR filtering, or passive stereo like the ZED family, and even iToF degrades under high ambient light from multipath interference. No calibration trick gets around this. Physics sets the limit, not engineering effort.

Indoor environments carry their own trap: polished floors, glass partitions, glossy tile. Active stereo's IR pattern distorts under specular and multipath reflection off these surfaces, and the resulting depth holes and disparity spikes don't disappear on their own. They persist as phantom obstacles sitting inside a robot's occupancy grid, wrong for as long as the lighting stays wrong. Research by Tsai on a RealSense D435 paired with a Jetson Orin Nano tackled this directly with a reliability-guided fusion approach, combining depth-reliability masking with region-guided fusion, running at 16.5 milliseconds. That result makes a specific point: the fix for this class of failure lives in the processing pipeline as much as in the sensor choice itself.

The lesson across both cases is the one that should have shaped the requirements worksheet from the start: no depth camera outperforms the environment it's asked to work in. Datasheet comparisons matter, but only once the deployment conditions get named honestly and specifically, not assumed away because the spec sheet looked good in a conference room.