The act of grasping is a step robot policies most often get wrong despite it having outsized importance on task success.
Robots performing tasks need to carefully measure how far open their grippers are at every moment; however, the human video used to train the robot provides no data like it.
In one published benchmark of vision-language-action models [1], a misaligned grasp caused 32% of 180 annotated failures in real and simulated trials, more than any other cause. In a second benchmark [2] on the low-cost SO-101 arm, grasp instability appeared in more than 90% of failed trials for each of the four policies tested, π0.5 and SmolVLA among them.
We believe egocentric datasets need to provide finer detail and labelings on the act of grasping.
We built a dataset
We collected an egocentric cooking dataset of our own to fill in the gaps of grasps in existing data.
Each take is cut into short subtask segments, about 28 for every minute of footage. Each segment names a hand and one of twenty-four actions the hands are doing: grasp, lift, transport, place, release, hold, cut, stir, etc.
The following table shows the share of frames by what either hand is doing. A frame counts towards a family of actions if either hand is inside a segment of that family, so the shares below do not add up to one hundred.
| Tool or ingredient work | 73.7% |
| Holding or steadying | 65.4% |
| Lifting, carrying or placing | 12.2% |
| Grasping | 4.1% |
Our labeling says what the hands are doing in almost every frame. They cover 98.8% of frames, with both hands labelled in 83.4%.
We have not found another public egocentric dataset that publishes hand-action breakdowns and labelings.
Our takeaways
Grasps are frequent but brief. A hand grasps something every 15 s on average, and together the grasps cover 4.1% of frames. However, a model trained on typical egocentric footage like this sees far too much holding than grasping for its importance on trial success.
The next addition to our dataset is a “grip layer”. This layer aims to augment human training video with grip signals that a robot’s gripper can leverage. We believe we can build simply from labels rather than from expensive hand tracking. A label does not need the fingers to be visible, and at the moment a hand closes, the object often hides them anyways.
The grip layer will contain:
- Grip state per frame. Closed, a labelled grasp to its release, Open otherwise.
- Grasp phase per frame. Reach, pre-shape, close, hold, release and retreat, mapped onto the actions we already label.
- Grip type per grasp. The 33 grasp types of the GRASP taxonomy [3], collapsed to seven coarse classes: cylindrical power, spherical power, pinch, tripod, lateral, hook and non-prehensile.
- Grasp timing to the frame. The closing frame itself, marked for every grasp.
The Yale Human Grasping Dataset [4] labelled grip types on head-camera footage all the way back in 2015, using an earlier version of the same taxonomy. We plan to add grip state and phase, aligned frame by frame with hand pose inside the training export. The first test is to label the held-out takes of our exam this way and split the exam’s error by phase.
To discuss the grip layer, get in touch at [email protected]
Sources
[1] Zhang, Y., Qi, Y., & Zheng, X. (2025). Experiences from benchmarking vision-language-action models for robotic manipulation (arXiv:2511.11298). arXiv. https://doi.org/10.48550/arXiv.2511.11298
[2] Yu, Y., & Qiu, X. (2026). Benchmarking vision-language-action models on SO-101: Failure and recovery analysis (arXiv:2606.08881). arXiv. https://doi.org/10.48550/arXiv.2606.08881
[3] Feix, T., Romero, J., Schmiedmayer, H.-B., Dollar, A. M., & Kragic, D. (2016). The GRASP taxonomy of human grasp types. IEEE Transactions on Human-Machine Systems, 46(1), 66–77. https://doi.org/10.1109/THMS.2015.2470657
[4] Bullock, I. M., Feix, T., & Dollar, A. M. (2015). The Yale human grasping dataset: Grasp, object, and task data in household and machine shop environments. The International Journal of Robotics Research, 34(3), 251–255. https://doi.org/10.1177/0278364914555720
The subtask labels and hand tracking are our own.