Whiteboard wiping
Misgrasping · residual ink · sticking · missed release
PCTac: Direct Structural-Color
Tactile Feedback for
Dexterous Manipulation
*Equal contribution.
†Corresponding authors.
1School of Information Science and Technology, ShanghaiTech University, Shanghai, China. {zhangr12023, liuxsh2025, daijzh2023, wukf, qinxh2024, quyr, xiaochx}@shanghaitech.edu.cn
2State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China. {liwanlin, jiaoziyuan}@bigai.ai
Fabrication
PCTac Fabrication
Force-response experiment
Optical response during the recorded loading sequence.
Optical response · cropped ROI
Original speed · cropped ROIQualitative optical response in a cropped tactile ROI. This footage is not a force or pressure calibration.
Qualitative optical response from a cropped camera ROI. This video is not a force calibration, and no force, pressure or response-time measurement is inferred from it. Color changes are not reconstructed physical height, depth or pressure.
Contact appearance
Eight objects. Distinct contact patterns.
Individual object photographs paired with their recorded tactile images.
Ping-pong
Object
Raw tactile image
Battery
Object
Raw tactile image
Peanut
Object
Raw tactile image
Heat sink
Object
Raw tactile image
LEGO
Object
Raw tactile image
Printed PLA
Object
Raw tactile image
Screw
Object
Raw tactile image
PCB
Object
Raw tactile image
Individual images extracted from the latest locally available source figure, without manuscript panel letters. Tactile images show optical appearance, not calibrated depth, force or pressure.
Embodied tasks
Real-world demonstrations of whiteboard wiping and test-tube insertion, including perturbed insertion.
Whiteboard wiping
Test-tube insertion
Test-tube perturbed
Manuscript-reported results
Successes / 30 trials per task. Selected systems are shown; complete source counts remain available in the downloadable data.
| Policy / tactile input | Whiteboard Wiping | Test-Tube Insertion | Perturbed Test-Tube Insertion | Overall |
|---|---|---|---|---|
| Vision-only | 19/30 (63.3%) | 9/30 (30.0%) | 1/30 (3.3%) | 29/90 (32.2%) |
| Marker-based Wedge RGB | 23/30 (76.7%) | 21/30 (70.0%) | 13/30 (43.3%) | 57/90 (63.3%) |
| PCTac RGB | 22/30 (73.3%) | 25/30 (83.3%) | 13/30 (43.3%) | 60/90 (66.7%) |
| PCTac Hue | 24/30 (80.0%) | 28/30 (93.3%) | 16/30 (53.3%) | 68/90 (75.6%) |
Where the systems fail
Failure modes remain part of the record, alongside selected demonstrations.
Test-tube insertion
Misgrasping · slipping · wedging · misinsertion
Perturbed insertion
Misgrasping · stalling · sticking · overstretching
Resources
Project overview
PCTac observes a continuous structural-color field at a photonic-crystal fingertip. Contact-induced appearance changes are represented as reference-relative circular Hue, without interpreting them as calibrated force or pressure. Tactile observations, external RGB views and robot state are combined in a robot-learning system.
The fingertip combines a photonic-crystal contact layer, internal white illumination and a folded optical path. One fingertip carries the sensor; the opposing contact surface is passive.