Awesome-Minecraft-Agents

Our Minecraft Agent

Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

We propose a Hybrid Multimodal Memory module that integrates structured knowledge and multimodal experiences into the memory mechanism of the agent. On top of it, we introduce a powerful Minecraft agent, Optimus-1, which achieves a 30% improvement over existing agents on 67 long-horizon tasks. ✨

Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy

We propose agent Optimus-2 which incorporates a Multimodal Large Language Model for high-level planning, alongside a Goal-Observation-Action Conditioned Policy (GOAP) for low-level control. Optimus-2 exhibits superior performance across atomic tasks, long-horizon tasks, and open-ended instruction tasks in Minecraft. ✨

Optimus-3: Towards Generalist Multimodal Minecraft Agents with Scalable Task Experts

We propose generalist agent, Optimus-3, endowed with multidimensional capabilities including Captioning, Embodied QA, Planning, Action, Grounding, and Reflection. Our comprehensive evaluation demonstrates that it consistently surpasses existing agents in the Minecraft environment across all assessed dimensions. ✨

---

Table of Contents

• Awesome Policy

  • Visuomotor Policy
  • Goal-conditioned Policy

• Awesome Agent

  • End-to-end Architecture
  • Hierarchical Architecture
  • Code based Agent

• Benchmark

---

Awesome Policy

Visuomotor Policy

Goal-conditioned Policy

---

Awesome Agent

End-to-end Architecture

Hierarchical Architecture

Code-based Agent

Benchmark