Sim-to-Real Transfer Techniques for Manipulation Policies

Sim-to-real transfer for manipulation policies depends on closing the right gap. Visual differences, physics errors, sensor noise, and missing deployment scenarios have distinct causes. A reliable transfer pipeline diagnoses those causes separately and matches each to a technique that can address it.

Why the sim-to-real gap is not one problem but four

A policy works perfectly in simulation and fails on the real robot. The natural debugging response is to improve the simulator, but that instruction is too broad to guide useful work. It collapses differences in appearance, dynamics, sensing, and scenario coverage into one problem. A sharper diagnosis separates four gaps, which are the visual domain gap, physics approximation error, sensor noise mismatch, and long-tail scenario absence. Each describes a different way that training can misrepresent deployment.

The visual gap comes from differences in texture, lighting, and color statistics between rendered images and real camera images. The physics gap concerns approximations of contact, friction, mass, and damping that diverge from real object behavior. The sensor gap concerns mismatched noise distributions in cameras, encoders, and force-torque sensors. The long-tail gap concerns situations absent from the training distribution. These distinctions matter because a policy can receive realistic images and still fail when contact forces diverge, so progress on one gap does not establish readiness across the others.

Why a single simulator's assumptions introduce bias that parameter tuning cannot remove

A friction value of 0.4 instead of 0.35 is a parameter error that system identification can correct. A simplified contact model that cannot represent soft-body deformation is a structural error that parameter fitting cannot correct. Both can produce a failed interaction, but only the first can be solved by finding better values within the existing model.

Simulator inductive bias is the set of assumptions and limitations built into an engine’s treatment of contact, deformation, and actuation. Whole-body-control policies exhibit discrepancies both across simulators and between simulation and the real world because those assumptions differ. For plush toy packing or rope routing, tuning friction and mass cannot compensate for a model that poorly approximates deformation. Higher fidelity can narrow this mismatch, but it does not eliminate assumptions and brings greater compute cost. This leaves a separate strategy, namely varying the simulator itself so that training does not depend entirely on one engine’s account of physics.

What domain randomization actually fixes and where it stops working

Varying textures, lighting, camera position, and color statistics encourages features that remain useful across visual conditions. Physics randomization does the same for mass, friction, and damping, but the ranges need to be based on real-world measurements. The practical discipline is therefore to randomize visual parameters aggressively and physics parameters conservatively around measured values. This makes domain randomization a strong visual technique and a more constrained dynamics technique.

Josifovski et al. established the central trade-off experimentally in their 2022 paper. More randomization helped transfer while making it harder for the policy to find a good solution in simulation. Too little variation leaves a brittle policy, while too much can produce robustness at the expense of precision. That compromise may be acceptable for gross pick-and-place but not for insertion or assembly. Choosing the range is also circular because covering real conditions requires some knowledge of those conditions.

The ceiling is not merely a matter of choosing better ranges. Randomizing parameters inside a flawed contact model never produces a different contact model, and sampling a distribution never exposes the policy to scenarios outside it. Domain randomization is therefore the right tool for the visual gap and a partial tool for physics. Sensor mismatch needs a fix that suits the sensor, while structural simulator bias and missing long-tail scenarios each need a different approach.

Closing the visual gap through photorealistic rendering rather than randomization

Photorealistic rendering takes the opposite approach to visual transfer. Instead of training a policy to tolerate differences in appearance, it reduces those differences at their source. If simulated imagery has the same relevant statistics as real camera imagery, there is less visual variation for the policy to overcome. RL-GSBridge, accepted at ICRA 2025, incorporated 3D Gaussian Splatting into a conventional reinforcement learning simulation pipeline and enabled zero-shot transfer for vision-based deep reinforcement learning.

Zhang et al. extended this approach to deformable-object tasks in 2025. They constructed soft-body digital twins from real-world videos and used 3D Gaussian Splatting to render robots, objects, and environments with photorealistic fidelity. They tested it on plush toy packing and rope routing. These examples connect visual reconstruction to tasks where dynamics are also difficult, but realistic appearance does not by itself establish accurate deformation. How much rendering and physics each help still needs to be assessed separately.

RoboRender, from Stanford, NVIDIA, and Microsoft Research in October 2026, makes that separation explicit. It converts simulated trajectories into photorealistic RGB videos while leaving dynamics and contacts under simulator control. It was tested on real robots doing pick-and-place, articulated-object manipulation, and mobile manipulation. Style-Identified CycleGAN provides a style-transfer route to real-like imagery without scene reconstruction, while shPix2pix applies the related idea to tactile streams. These approaches broaden the options beyond building a photorealistic scene model.

Scene specificity remains an important limit for 3DGS reconstructions and digital twins. Building them for every deployment environment takes manual effort and restricts scale across diverse tasks. RoboRender and learned rendering bridges are not scene-specific in the same way because they learn a transform rather than reconstruct each scene. They still require real imagery for training, however. The relevant comparison is therefore between different real-data and reconstruction costs.

Closing the physics gap through system identification, digital twins, and multi-simulator training

Closing the dynamics gap requires matching real interactions. System identification starts with the least structural intervention by fitting parameters to the real system. It can correct mass, friction, and damping mismatches when the simulator’s underlying model is adequate. It cannot repair a model that lacks the required contact behavior, and it requires hardware access that may be unavailable during early training. Its value therefore depends on whether the task needs better calibration or a different representation of physics.

Torne et al. developed RialTo at MIT in 2024 to strengthen real-world imitation learning policies through reinforcement learning in digital twin environments. A scanning interface constructs those environments from small amounts of real-world data. An inverse distillation procedure brings real demonstrations into simulation, where the policy can be trained further. RialTo needs three days of training before it can be deployed. Its twin must also be rebuilt for each new scene, so better task alignment comes with a recurring preparation cost.

PolySim, from Lei et al., addresses structural bias at a different level. It trains policies jointly across heterogeneous simulators within a single training run. Multiple engines run environments in parallel, so the policy sees the same task modeled in different ways across simulators. The choice of simulator can then be randomized. This varies the dynamics in ways that sampling parameters within one engine cannot.

These distinctions matter most for contact-rich assembly, insertion, and deformable-object handling. These tasks rely heavily on contact behavior, and poor deformation models can mean that a lot of real demonstration data is needed. Scheikl et al.’s 2023 work on robot-assisted surgery places this concern in a safety-critical setting, where unsafe real training data cannot be collected as a routine remedy. System identification, digital twins, and multi-simulator training tackle different sources of dynamics error, so which works best depends on the source. Even a successful choice among them leaves sensor noise and long-tail coverage as separate problems.

Sensor noise mismatch and why tactile and proprioceptive transfer break differently from vision

Sensor mismatch changes the information a policy receives about its own state and actions as well as the appearance of its surroundings. Photorealistic pixels can still differ from real camera data because of temporal correlation, read noise, and motion blur. Force-torque sensors, joint encoders, and tactile arrays add hardware-specific noise characteristics that are difficult to measure analytically. In closed-loop contact-rich manipulation, this mismatch corrupts the feedback on which the policy depends. Simply adding randomized noise helps only to the extent that the chosen distribution resembles the real one.

Freud et al.’s August 2025 shPix2pix tactile GAN translates simulated tactile images into realistic equivalents with shear deformations. It targets the sensor stream directly rather than treating tactile transfer as a camera-rendering problem. Action abstraction offers a complementary mitigation by having the policy issue target joint angles in place of raw torque signals. This reduces the opportunity for low-level control noise to corrupt policy decisions. Sensing still matters with both approaches, which work on the feedback path.

The long-tail gap and why simulation cannot generate the scenarios that matter most at deployment

Simulation can reproduce only the scenarios its generation process permits. Domain randomization and high-fidelity simulation both remain bounded by designer-specified distributions, so greater fidelity does not reveal what those distributions omit. A cable unexpectedly draped across a workspace need not expose an incorrect physics model. It may expose a situation the training process never generated.

Better rendering, contact models, and sensor statistics can make familiar scenarios more faithful without introducing unfamiliar contingencies. Purely simulated training therefore leaves deployment failure modes that are structurally absent from its data. Real-world data remains necessary because of the limits of coverage: it can expose conditions that simulation designers did not encode.

How sim-and-real co-training addresses the long-tail gap without requiring large real datasets

Co-training combines simulation’s scale with real data that grounds the policy in deployment conditions. Maddukuri et al.’s RSS 2025 study found that a robot arm and a humanoid platform both performed much better in the real world when trained on a mix of simulation and real data than on limited real data alone. The study tested task-aware data from digital cousins and task-agnostic data from more diverse prior simulations. Both helped even though the simulated and real data differed a great deal. The result supports using simulation to supply transferable structure rather than demanding that it reproduce every deployment condition.

Within that recipe, Maddukuri et al. used CogVideoX to reduce visual mismatch. They fine-tuned the video diffusion model on real demonstration videos in the Text2Video modality. They then added noise to reference simulation videos and used those videos to initialize diffusion, preserving object positions and dynamics while improving visual realism. This rendering bridge was one ingredient in the co-training method. Lei et al.’s 2026 mechanistic analysis also found that co-training worked with small diffusion policy models and large Vision-Language-Action models.

Task-aware simulation data consistently outperformed task-agnostic data, showing that alignment still matters even when both sources help. A related pipeline works in stages, first pretraining in simulation, then collecting targeted real data and fine-tuning on it. Both approaches give real observations a defined role without requiring all training experience to come from hardware. The optimal ratio remains task-dependent, and a small real dataset addresses coverage without guaranteeing that every rare contingency has been captured.

Matching the right technique to the right gap for a given manipulation task

For quasi-static pick-and-place, the visual gap is usually the primary bottleneck, making domain randomization or photorealistic rendering the main intervention. Physics is secondary in this setting, and system identification usually suffices. For contact-rich assembly and insertion, the emphasis shifts toward digital twins or multi-simulator training because contact behavior governs success. Force-torque sensing also matters, which gives noise modeling and action abstraction a distinct role.

Deformable-object manipulation has both structural physics errors and large gaps in coverage. For cloth folding, rope routing, and surgical tissue, the evidence favors using multi-simulator training with real-data co-training over parameter tuning alone. Photorealistic soft-body digital twins can also help, but they still come with a cost for each scene. In robot-assisted surgery, sensor and long-tail gaps lower task performance and pose safety concerns. Real-world contingencies cannot be treated as covered simply because the simulated task succeeds.

The diagnostic habit is to identify the gap before adding the technique. Simulation pretraining, targeted real-data collection, and fine-tuning are common stages in a pipeline, but the work inside simulation should change with the task. Current evidence favors co-training for scale and digital twins for precision in known environments. It does not establish a universal balance between high-fidelity scene-specific training and broader, less aligned simulation. How much fidelity a task needs before greater coverage becomes the more valuable investment remains unresolved.