ScienceWorld

ScienceWorld is a text-based virtual environment centered around accomplishing tasks from the standardized elementary science curriculum. This code accompanies the paper ScienceWorld: Is your Textual Agent Smarter than a 5th grader?.

[!WARNING] Potentially breaking change in 1.3.0: Environment objects are now enumerated in deterministic UUID order across resets and Scala/JVM versions. This fixes stochastic simulation outcomes, but can change physics trajectories, ambiguous action resolution, gold paths, and observation/JSON ordering compared with 1.2.x. Task generation for a given variation is otherwise unchanged.

Demo and examples

You can try ScienceWorld yourself via our HuggingFace Space or read some of the playthrough transcripts.

Citation

@inproceedings{wang-etal-2022-scienceworld,
    title = "{S}cience{W}orld: Is your Agent Smarter than a 5th Grader?",
    author = "Wang, Ruoyao  and
      Jansen, Peter  and
      C{\^o}t{\'e}, Marc-Alexandre  and
      Ammanabrolu, Prithviraj",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-main.775/",
    doi = "10.18653/v1/2022.emnlp-main.775",
    pages = "11279--11298",
}

Quickstart

Before running: You will have to have Java 1.8+ installed on your system (shipped with most linux distributions) and Python 3.8+. We recommend creating a conda environment like this:

conda create --name scienceworld python=3.8
conda activate scienceworld

Then, install ScienceWorld either from PyPi:

pip install scienceworld

or from source in development mode:

git clone https://github.com/allenai/ScienceWorld.git
cd ScienceWorld
pip install .

Run an example random agent, on task 13 (classification: place a non-living thing in a box), for 5 episodes:

python examples/random_agent.py --task-num=13 --num-episodes=5 --simplifications-preset easy

Run a user console where you can interact with the environment, on task 3 (change of state: melting):

python examples/human.py --task-num=3 --num-episodes=5

Web Server Demo

A web server demo is also available, that allows running a ScienceWorld user console that can be interacted with in a web browser.

To run the web server demo:

conda create --name scienceworld python=3.8
conda activate scienceworld
pip install scienceworld[webserver]

Run the web server:

python examples/scienceworld-web-server-example.py

Point your web browser to: localhost:8080

ScienceWorld Design

ScienceWorld is written in Scala (2.12.9), and compiles using sbt into a JAR file that is run with Java. For convenience, a Python API is provided (Python >= 3.8), which interfaces using the py4j package.

If you modified the Scala code, you can recompile the JAR file by running:

./simulator/package.sh
pip install -e .

Tasks

The tasks are listed in the table below along with their number of variations. Either the task ID or its name can be used to a task with env.load().

Simplifications

ScienceWorld supports a number of simplifications that can be applied to the environment to make it easier for agents to learn. These simplifications can be applied by passing the --simplifications-preset argument to the command line interface, or by passing the simplifications argument to the Python API.

The available simplifications are:

• teleportAction: Allows agents to instantly move to any location in the environment.
• openDoors: All doors in the environment are open by default.
• selfWateringFlowerPots: Automatically waters all flower pots in the environment.
• noElectricalAction: Disables electrical actions, making it easier for agents to learn tasks that do not require electrical actions.
• openContainers: All containers in the environment are open by default.

The --simplifications-preset argument can be set to easy to apply the following simplifications:

• teleportAction
• openDoors
• selfWateringFlowerPots
• noElectricalAction (for non-connectivity tasks)

[!WARNING] The easy preset differs from what is described in the paper (see Appendix B.5). The openContainers is not included in that preset and should manually be added if desired.

Baseline Agents

DRRN: https://github.com/cognitiveailab/drrn-scienceworld

KG-A2C: https://github.com/cognitiveailab/kga2c-scienceworld

CALM: https://github.com/cognitiveailab/calm-scienceworld

Behavior Cloning and Decision Transformer: https://github.com/cognitiveailab/t5-scienceworld

Developers

To compile the ScienceWorld JAR file, follow these steps:

Prerequisites

You will need to have Java 1.8+ SDK installed on your system (shipped with most linux distributions). E.g. on Ubuntu, you can install it with:

sudo apt-get install openjdk-21-jdk

Then, install sbt (Scala Build Tool) by running:

    echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | sudo tee /etc/apt/sources.list.d/sbt.list
    echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | sudo tee /etc/apt/sources.list.d/sbt_old.list
    curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo tee /etc/apt/trusted.gpg.d/sbt.asc
    sudo apt-get update
    sudo apt-get install sbt

Building the JAR

Once you have sbt installed, you can compile the ScienceWorld JAR file by running:

./simulator/package.sh