Advancing Embodied AI with Real-World Egocentric Video Data
Enabling human activity recognition, robotics, & hand-object interaction models with scalable video captured in household and industrial environments.
Project Overview
The client partnered with Shaip to establish a scalable collection program for high-quality egocentric video data. Contributors would record genuine household and industrial activities using securely mounted mobile devices positioned at forehead level.
The objective was to build a diverse, naturalistic video dataset suitable for training models to:
- Recognize complex human activities and task sequences
- Interpret hand-object interactions
- Understand real-world household and workplace environments
- Support robotics and embodied AI applications
- Improve model generalization across locations, devices, objects, and operating conditions
The engagement emphasized natural task performance, stable camera positioning, continuous execution, contributor consent, and rigorous quality validation.
Egocentric Video Dataset: Key Specifications
Data Volume
30,000 hours collected, 24,000+ hours annotated
Contributor Capacity
Up to 100 approved video hours per participant
Approved Devices
iPhone 11+, Google Pixel 6+, & Samsung Galaxy S21+
Geographic Coverage
United States and Latin America
Activity Environments
Household, industrial, & commercial settings
Challenges in Egocentric Video Data Collection
- Capturing Natural Human Behavior: Contributors needed to perform genuine activities without staged, exaggerated, or robotic movements.
- Maintaining First-Person Framing: Both hands, the working area, and relevant objects had to remain visible throughout active task execution.
- Controlling Camera Stability: Head-mounted mobile devices needed to remain secure with minimal wobble, excessive tilt, or obstruction.
- Collecting Across Diverse Environments: The dataset required authentic footage from residential, industrial, retail, hospitality, agricultural, construction, and service environments.
- Minimizing Idle or Non-Task Footage: Recordings had to demonstrate continuous, intentional work with limited pauses and unnecessary movement.
- Ensuring Authorization and Consent: Every contributor and recording location required appropriate permission, consent, and compliance validation.
- Scaling Without Compromising Quality: Large volumes of footage needed to be reviewed quickly, with clear rejection reasons and efficient recapture workflows.
How Shaip Collected the Egocentric Video Data
Collection Strategy
Shaip designed a flexible framework covering authentic household, commercial and industrial activities. Contributors selected tasks aligned with their natural routines rather than following staged scripts.
Contributor Onboarding
Participants across the United States and Latin America were trained on task selection, device requirements, camera placement, consent and prohibited capture conditions.
Mobile Video Collection
Recordings were captured through a designated mobile application with automated submission and location verification to confirm capture within approved regions.
Head-Mounted Camera Setup
Approved smartphones were securely mounted at forehead level and angled approximately 45° downward to keep the contributor’s hands, objects, workspace and, when stationary, toes visible.
Natural Task Execution
Contributors performed continuous, purposeful activities while standing or walking and interacting naturally with tools, equipment and everyday objects.
Quality Validation
Every recording was checked for camera stability, lighting, focus, hand-object visibility, task continuity, posture, environment and authorization.
Rapid Review and Recapture
Submissions were reviewed within one day. Clear rejection reasons enabled contributors to correct and recapture footage quickly.
Governance Controls
Consent, location authorization, secure submission, location validation & intellectual-property assignment were integrated into the workflow.
Project Scope & Activities
| Data Segment | Representative Activities | Capture Requirements |
|---|---|---|
| Household Activities | Laundry, room organization, kitchen tidy-up, pet care, plant care, gardening, and home maintenance. | Standing or walking; natural execution; hands and workspace visible. |
| Warehouse and Logistics | Packaging, sorting, inventory handling, inspection, and material movement. | Stable head-mounted capture with continuous task activity. |
| Commercial & Hospitality | Janitorial work, housekeeping, retail stocking, and food-service back-of-house activities. | Authorized recording location and clear object interaction. |
| Industrial Activities | Light manufacturing, assembly, equipment maintenance, and production processes. | Genuine workplace execution with appropriate permissions. |
| Agriculture and Landscaping | Grounds maintenance, gardening, agricultural work, and outdoor material handling. | Adequate lighting, stable framing, and safe task execution. |
| Skilled Trades & Field Services | Construction, automotive service, repairs, installation, and field maintenance. | Clear visibility of tools, hands, components, and work area. |
Quality Framework
Recordings were rejected for:
- Obstructed, excessively tilted, or unstable camera views
- Low lighting, weak focus, unclear visibility
- Long idle periods or non-task footage
- Artificial, contrived, or robotic execution
- Excessive head, hand, or body movements
- Hands outside the frame for more than 10% of active grab-and-release activity
- Unauthorized recording locations
- Missing contributor consent
A successful recording was required to:
- Capture authentic household or industrial work
- Maintain a stable first-person perspective
- Keep hands, objects, and the task area visible
- Demonstrate continuous and intentional activity
- Meet approved device and camera-positioning requirements
- Be suitable for AI training involving activity recognition and robotic perception
Outcomes: Robotics-Ready Egocentric Video Data
- Authentic Activity Data: Natural task execution captured human actions, hand-object interactions and operational sequences.
- Scalable Dataset Expansion: Each participant could contribute up to 100 approved hours, while variations in objects, tools, locations, lighting and task sequences expanded dataset diversity, scaling the program to an estimated 30,000 hours of egocentric video collected, with 24,000+ hours annotated and approved for model training.
- Robotics-Ready Context: Stable first-person footage provided clear visibility of hands, objects, equipment, workspaces and task progression.
- Rapid Quality Intervention: One-day review cycles enabled fast feedback, correction & recapture.
- Geographic and Device Diversity: Collection across the USA and Latin America, using approved Apple, Google and Samsung devices, introduced practical environmental and hardware variation.
- Integrated Governance: Consent, authorization, location validation, secure submission and qualitycontrols were embedded throughout the program.
The resulting collection approach provides a strong foundation for building AI systems that understands human action from an operator’s perspective and function more reliably
Shaip’s structured approach to egocentric video collection gave us the scale, quality controls, and real-world diversity required for our AI training program. Their rapid review workflow and detailed capture standards created a reliable foundation for advancing activity-recognition and robotics models.
