Sample project
Overhead kitchen task: 20 object tracks, hand contacts and action captions
- Data
- Video · 1920×1080
- Volume
- 150 frames · 2,711 labeled instances
- Status
- Sample project
The problem
A robot learning a kitchen task needs to know what every object is, where it goes from frame to frame, when a hand touches it, and which step of the task is happening. Raw video has none of that.
What we did
- Object inventory of 19 classes, from knife and peeler to recipe card
- Anchor boxes placed by a reviewer, then propagated with SAM 2.1 video segmentation
- 20 tracks reviewed on per-object contact sheets; drifting tracks corrected with extra anchors
- 23 hand-object contact events derived from mask overlap and reviewed
- 10 timestamped action captions plus scene captions
- Exported as COCO (with outlines), YOLO, CVAT video XML and WebVTT
Result
150 frames with 2,711 labeled instances across 20 tracks, 23 hand-object contacts and 12 captions, delivered in four formats.
Labeled video · 30 seconds
- Caption track
- 00:00Yellow watermelon wedges are cut into cubes on the cutting board.
- 00:04A glass mixing bowl is set on the board and the watermelon cubes are moved into it.
- 00:08The knife is set down and the cucumber is brought to the board.
- 00:09The cucumber is peeled with the red peeler.
- 00:11The peeled cucumber is halved lengthwise and diced with the knife.
- 00:15The diced cucumber is added to the mixing bowl and the last pieces are cleared from the board.
- 00:16The board is wiped with a paper towel.
- 00:17The red onion is picked up, peeled and cut in half.
- 00:20The onion half is sliced and finely chopped with the knife.
- 00:29Chopped onion is scraped off the board into the mixing bowl with the knife.
Frames, depth and review sheet
Sample output
An excerpt from the delivered files. "source": "ai" marks boxes propagated by the segmentation model; every one was reviewed by a person before export.
{
"image_id": 55,
"category": "olive oil bottle",
"bbox": [63, 10, 273, 209],
"attributes": {
"track_id": "obj_5fe3a3683a8a",
"occluded": false,
"keyframe": true,
"source": "ai"
},
"segmentation": [[69, 29, 64, 40, 65, 58, 90, 88, "…"]]
}Source footage: “Farmers Market Food Demo - Watermelon Salad”, US Department of Agriculture (Agricultural Marketing Service), via Wikimedia Commons (Public domain). Labels, captions, tracks and depth by Annoroid.