School project at Purdue University

Vision-guided autonomous wound scanning and treatment

A UR16e finds wounds on a patient by itself, measures them in 3D, writes a clinical report, and holds a perpendicular treatment pose while the patient moves.

Project developed with Praveen Krisna

3 min read

Shipped
UR16e with a wrist-mounted Kinect scanning a mannequin in the lab, live skeleton and wound detections alongside

Abstract

A robot that autonomously locates, records, and treats wounds. First, it detects the patient's pose and limb locations to establish a scanning route. Sweeping an arc 50 cm above the patient, it measures wounds in 3D and records data like size, classification, and body location. For the simulated treatment phase, it approaches to 15 cm. Because patients move, it activates a locking mechanism. This tracks motion to hold a specific distance and stay normal to the wound surface.

Introduction

Clinicians visually estimate wound size and severity today. The process is subjective, slow, and hard to reproduce. In hazardous environments like radiation zones, manual checks put operators at risk. This project automates assessment for a reliable baseline.

Two pipeline phases:

  • Scanning. Find every wound from a standoff distance, extract its 3D centroid and normal, and output a clinical report.
  • Locking. Maintain a perpendicular 15 cm standoff as the patient moves, without overloading the motion planner.
Block diagram of two ROS 2 pipelines, scanning and locking, sharing one camera topic namespace
Figure 1. Two launchable pipelines over one hardware interface. Both publish the same camera topics, so the simulated locking nodes run unchanged on the robot once the bridge is swapped out.

Calibration: mapping pixels to reality

Accurate spatial tracking demands precise calibration.

  • Intrinsics. A 9x6 checkerboard and OpenCV calibrateCamera yield the camera matrix and distortion values.
  • Depth registration. Nearest-neighbor interpolation only. Bilinear averaging blends uint16 depth across edges and ruins the plane fit.
  • Eye-in-hand. We solved AX = XB over 20 poses using Tsai's method to find the camera-to-gripper transform.

Viewpoint planning

Camera arc of candidate viewpoints around a torso centre, with three selected scan poses
Figure 3. Arc sweep around the torso centre, filtered by IK feasibility.

A single frontal view misses peripheral wounds. The camera instead sweeps an arc around the patient.

  • Candidate filtering. Viewpoints pass UR16e inverse kinematics and are scored by a manipulability proxy.
  • Path validation. The chosen three poses are jointly validated by simulating their joint-space path.

Detection and surface geometry

YOLO wound detection, the deprojected depth patch, and the fitted surface normal in RViz
Figure 2. Box, point cloud, fitted plane, normal.

A YOLOv8 model fine-tuned on 140 images yields 2D boxes. The rest is geometry.

  • Deprojection. Box pixels project into 3D using the pinhole model and registered depth.
  • Outlier rejection. Points outside two standard deviations from the patch median are dropped.
  • Normal estimation. Singular Value Decomposition (SVD) on the centered cloud gives the surface normal.
centered = points - points.mean(axis=0)
normal = np.linalg.svd(centered, full_matrices=False)[2][-1]
normal = -normal if normal[2] > 0 else normal

This normal defines the approach axis. Roll remains free. We project the X-axis onto the target plane to stop wrist spin.

From pixels to a clinical report

Clip 1. Scanning run on the physical robot.
Generated clinical report entry with wound crop, location, depth, size, redness and robot pose
Figure 4. One entry per unique wound: position, body part, redness, confidence.

The multi-view arc sees the same wounds repeatedly.

  • Deduplication. Detections merge in 3D. Centroids within 8 cm are duplicates.
  • Anatomical labeling. Wound location maps to the nearest of 32 skeleton bone segments.
  • Severity scoring. A redness index over the cropped image serves as a severity proxy.

Holding the pose

Clip 2. Locking and drift correction in Gazebo.
  • Smoothing. A sliding window outputs median position and SLERP-averaged orientation.
  • Deadband control. Corrections only trigger past 1.5 cm or 8.6 degrees of drift.
  • Execution. The Pilz motion planner handles corrections in Point-to-Point (PTP) mode.

Results

Metric Value
mAP@50 73%
Precision, recall 0.88, 0.63
Wounds found 40 of 56
Correct body part 75.6%
Single-view baseline 64.3%
Grid of validation images comparing ground-truth wound labels against model predictions
Figure 6. Labels against predictions. The model is conservative.
  • The detector rarely hallucinates, but recall is bottlenecked by the small dataset size.
  • The three-pose arc sweep beats a single center view by 7 percentage points.
  • Localisation accuracy surpasses detection rate. Failures are mostly missed detections, not mislabeled ones.

Limits and next steps

  • Due to the Kinect V2's 50 cm minimum range, we could not test the 15 cm treatment step on the physical robot. It was only tested in Gazebo simulation.
  • Recall requires a larger dataset stratified across skin tones and wound types.
  • Next steps: deploy active locking on physical hardware and add a severity-driven treatment dispenser.