Meta and Tesla Use Human Data to Train AI Agents

Meta's Model Capability Initiative (MCI) / Agent Transformation Accelerator (ATA):

• Software installed on US employee work computers.
• Tracks mouse movements, clicks, and keyboard inputs across work-related applications and websites.
• Periodically captures screenshots of employee work content.
• Data is intended strictly for training AI models to understand human-computer interactions and is not for performance reviews.

Tesla Optimus AI Training:

• Utilizes first-person video data from human task demonstrations, captured by camera rigs (helmet + backpack with 5 in-house cameras).
• Incorporates data from Tesla's fleet of millions of vehicles for visual and spatial understanding, which transfers to robot cognition.
• Employs a 'world simulator' that generates over 10,000 synthetic training variations per demonstrated task.
• All data streams converge on 'Cortex,' a computing cluster with over 67,000 H100-equivalent GPUs.
• Neural networks train in approximately 70,000 GPU hours per complete cycle.
• Uses a unified neural architecture shared with Full Self-Driving (FSD) systems.

Figure AI's Helix Model and Training:

• Helix Lab: A dedicated research and development facility for large-scale data collection.
• Focuses on capturing extensive 'egocentric human video' and interaction data from real-world environments.
• Helix Vision-Language-Action (VLA) model: Directly controls Figure's humanoid robots.
• Achieves 'zero-shot human video-to-robot transfer,' meaning robots learn from human videos without explicit step-by-step programming.
• Employs Reinforcement Learning (RL) in high-fidelity physics simulations.
• Thousands of virtual Figure 02 robots are simulated in parallel, collecting years of data in a few hours.
• Uses 'domain randomization' in simulation combined with high-frequency torque feedback for 'sim-to-real transfer' without additional tuning.
• Rewards the robot for mimicking human walking reference trajectories to achieve human-like gait.