Internship at Magna International, Advanced Robotics R&D

Dual-arm assembly R&D: from CAD to a learned insert

Research and development internship at Magna. A dual-arm pipeline in Isaac Sim that plans a full production assembly in 59 seconds, then hands the last millimeters of insertion to a learned policy.

3 min read

Shipped
Two six-axis arms on a shared column above a work table, each fitted with a gripper

Abstract

Research internship at Magna Advanced Robotics. I built a dual-arm pipeline in Isaac Sim that takes production CAD models and plans a full robotic assembly in just 59 seconds. Classical geometry handles the gross movements, but provably fails on the final millimeter of insertion due to physical interference. To solve this, a machine learning policy takes over for the last few centimeters, seamlessly integrated into a single execution task.

Introduction

Geometric path planning solves how a robot moves through free space. It completely fails when parts actually touch. Three observations drove this project:

  • Physics matters. A trajectory that passes every collision check can still jam under physical grasping constraints because the planner lacks a contact model.
  • Micro-tolerances. One part could not be inserted at all using classical planning. A residual tilt of just 2.2 degrees caused the arm to press to 300 Newtons without seating.
  • Interference fits. Another component seats through roughly a millimeter of actual physical interference. This is a corridor narrower than the inherent positional scatter of any AI policy trying to learn it.

The architecture is built around this boundary: everything before contact stays classical, and machine learning is only spent on the final insertion.

The pipeline

A geometric chain from CAD to physics replay, which records a dataset, which trains a learned insert that re-enters the plan as an operator. GEOMETRIC, DETERMINISTICCADproduction assemblyAssembly planextraction pathsTask plan31 segmentsMotion plancuRobo, 12-DOFPhysics replayPhysX checkWHERE GEOMETRY STOPSLEARNING IS SPENT HEREOnly on the last few centimetresof each insertion.Learned insertRL, VLA, residualDatasetLeRobot, 25 Hzre-enters as an operator
Figure 2. One chain with a closing edge. Everything above the line is geometric and deterministic.
  • Geometric planning. The assembly planner searches for extraction paths using pure geometry. Threaded and press-fit parts require screw motions to unbind.Rotating while sliding unbinds what pure sliding binds.
  • Task expansion. The task planner expands the high-level order of work into 31 distinct segments, statically validating them in under a second without running the physics simulator.
  • Data generation. The physics replay serves as both a plan checker and a dataset recorder. Every recorded demonstration is guaranteed to be an executable path.
  • Drop-in learning. A trained policy is not a separate script. It runs as a standard pipeline operator, meaning a single config change swaps a classical insertion for a learned one.

Planning results

Metric Performance
Segments planned 31 of 31
Full assembly time 59.24 s (1.31 s variance across runs)
Median segment time 1.32 s
Speedup 4.4x faster than the previous engine
  • Coordinated movement. The dual-arm carry is the most expensive operation, taking up 24% of the time despite being less than 10% of the plan.
  • Cost drivers. Execution time is driven by physical path length, not the planning mode.
  • Seamless integration. Routing the final insertion through a learned policy cost no more execution time than the classical equivalent.

Learned skills results

Instead of human teleoperation, a privileged expert script combined with the planner generated exactly 150 successful demonstrations (mean seat error 1.005 mm) under full randomization.

  • Reinforcement learning. Achieved a 60% success rate at 9.7 N median peak force, beating a scripted baseline (71% success, but a massive 47 N peak force). Success depended heavily on curating the demonstrations, not the algorithm.
  • Vision-Language-Action (VLA). Closed-loop vision successfully seated parts where open-loop movements failed (3 of 5 success rate). However, when retrained with real physical contact data, success dropped to zero because the policy's positional scatter was wider than the physical clearance corridor.
  • Residual models. 11 of 11 parts seated perfectly when the exact mathematical correction was explicitly supplied. But when left to learn the correction across four models, it achieved 0 of 436. Proprioception alone cannot "see" the tiny errors the residual is meant to fix.
Bar charts comparing seat success for open-loop and closed-loop policies across randomization scales, and against a privileged expert
Figure 3. What closed-loop vision buys, and where retraining on real contact does not.

Limits and next steps

  • Hardware bottleneck. Training throughput is severely capped by having only one GPU and one simulation environment.
  • Memory constraints. The residual model desperately needs visual input to succeed, but the frozen base model and the simulator leave no VRAM left for a vision encoder.
  • Sim-to-real gap. The force-feedback teleoperation track remains an analysis project. Nothing has been deployed to physical hardware yet.