Learning to play Minecraft with Video PreTraining

OpenAI has published a research paper on Video PreTraining (VPT), a method for training neural networks to play Minecraft using a large dataset of unlabeled human gameplay videos. The model learns from raw keyboard and mouse inputs, requiring only a small amount of labeled contractor data for fine-tuning. With this approach, the agent can craft diamond tools—a task that typically takes experienced players over 20 minutes and 24,000 actions. VPT represents a significant advance in training agents that interact with software through native human interfaces, moving toward general-purpose computer-using AI. For developers building AI workflows, this research demonstrates how unlabeled video data can dramatically reduce the need for costly labeled datasets when training complex behavioral models. The technique could be applied to other domains where human demonstration videos are abundant, enabling more efficient development of autonomous agents for tasks like software testing, data entry, or game automation.