Dexterous manipulation with full-hand tactile sensing

Abstract

Graduate research at the Purdue Mars Lab focusing on dexterous manipulation using a 16-degree-of-freedom LEAP Hand. Vision fails when a robotic hand closes over an object, so we rely on tactile sensing instead. My role centers on hardware and software design: I create manipulation tasks that require touch, model 3D test fixtures for the sensors, and build sensor fusion systems to enable contact-rich teleoperation.

Introduction

When a robotic hand grabs an object, it blocks its own view. At the exact moment contact matters most, cameras become useless. Touch remains the only reliable sense, which is why our lab focuses on manipulation driven by physical contact rather than vision.

This research aims to answer several open questions:

• Task relevance. Which tasks actually require a sense of touch instead of just better cameras?
• Sensor validation. How do you accurately test and characterize a tactile sensor before trusting it on a moving robot?
• Teleoperation. How do you combine tracking inputs so a human operator can actually feel their way through a task?

The sections below detail the manipulation task design, the physical test fixtures, the teleoperation pipeline, and my research into tactile encoders.

Task design

A task only matters if physical contact carries crucial information. If vision alone can solve it, it proves nothing about our tactile sensors.

• Focusing on touch. I design the physical manipulation tasks used to evaluate the robotic hand.
• Driving the data. The choice of task dictates the quality of the recorded demonstrations, so task design must happen before any data is collected.

Physical test fixtures

Characterizing a brand-new sensor directly on a moving robotic hand introduces too many variables. Testing must happen in a controlled, isolated environment first.

• CAD modeling. I design test fixtures in Fusion 360 that hold sensors in highly repeatable positions to measure their response.
• Rapid iteration. Multi-fingered manipulation requires multiple mounting angles and geometries. Every time a sensor is revised, the fixtures must be redesigned to match.

Teleoperation architecture

We use two tracking sources to control the robot. An Apple Vision Pro headset tracks the operator's wrist pose and provides a first-person view. Rokoko motion-capture gloves track individual finger movements.

• Sensor fusion. I explore how to fuse these two distinct tracking streams into a single, cohesive command for the robotic hand.
• Kinematic retargeting. A human hand and the 16-DOF LEAP Hand do not move the same way. The operator's motion must be algorithmically translated to preserve the intent of the grasp, rather than just copying raw joint angles.Toggle noteRetargeting maps a human pose onto a different mechanical joint structure while preserving the functional goal, like pinching or wrapping.
• Contact-rich control. The ultimate goal is to feed tactile signals back into the operator's decision loop in real time.

Limits and next steps

This is active research, currently focused on understanding how tactile data is best processed.

• Encoder research. I am surveying how raw tactile signals are transformed into mathematical representations that a robotic policy can actually use.
• Informing the fusion. While I do not build the neural encoders myself, this research directly informs how I design the teleoperation and sensor fusion pipelines.