Your browser does not support the video tag.

[!IMPORTANT] WebLINX is now available as a benchmark through BrowserGym, allowing you to access demonstration steps in the same way you would access a web agent environment like WebArena or MiniWoB. This also allows you to run agents from the Agentlab library, including agents that achieve SOTA performance through Claude-3.5-Sonnet. To enable this integration, we are releasing the weblinx-browsergym extension for BrowserGym on PyPi, as well as a new dataset, WebLINX 1.1, derived from WebLINX on Huggingface. In WebLINX 1.1, a small number of demonstrations were removed after processing, but no new demonstration was added. There are substantial changes to the steps being evaluated, with the inclusion of tab actions. Please report your results as "WebLINX-1.1", "WebLINX-BrowserGym" or "WebLINX-BG" in your work, to differentiate from the initial release of weblinx (1.0).

Quickstart

To get started, simply install datasets with pip install datasets and load the chat data splits:

from datasets import load_dataset
from huggingface_hub import snapshot_download

# Load the validation split
valid = load_dataset("McGill-NLP/weblinx", split="validation")

# Download the input templates and use the LLaMA one
snapshot_download(
    "McGill-NLP/WebLINX", repo_type="dataset", allow_patterns="templates/*", local_dir="."
)
with open('templates/llama.txt') as f:
    template = f.read()

# To get the input text, simply pass a turn from the valid split to the template
turn = valid[0]
turn_text = template.format(**turn)

You can now use turn_text as an input to LLaMA-style models. For example, you can use Sheared-LLaMA:

from transformers import pipeline

action_model = pipeline(
    model="McGill-NLP/Sheared-LLaMA-2.7B-weblinx", device=0, torch_dtype='auto'
)
out = action_model(turn_text, return_full_text=False, max_new_tokens=64, truncation=True)
pred = out[0]['generated_text']

print("Ref:", turn["action"])
print("Pred:", pred)

Raw Data

To use the raw data, you will need to use the huggingface_hub:

from huggingface_hub import snapshot_download

# If you want to download the complete dataset (may take a while!)
snapshot_download(repo_id="McGill-NLP/WebLINX-full", repo_type="dataset", local_dir="./wl_data")

# You can download specific demos, for example
demo_names = ['saabwsg', 'ygprzve', 'iqaazif']  # 3 random demo from valid
patterns = [f"demonstrations/{name}/*" for name in demo_names]
snapshot_download(
    repo_id="McGill-NLP/WebLINX-full", repo_type="dataset", local_dir="./wl_data", allow_patterns=patterns
)

For more information on how to use this data using our official library, please refer to the WebLINX documentation.

Reranking Data

You can also access the data processed for reranking tasks. To do that:

from datasets import load_dataset

path = 'McGill-NLP/WebLINX'

# validation split:
valid = load_dataset(path=path, name='reranking', split='validation')
# test-iid split
test_iid = load_dataset(path, 'reranking', split='test_iid')
# other options: test_cat, test_geo, test_vis, test_web

print("Query:")
print(valid[0]['query'])

print("\nPositive:")
print(valid[0]['positives'][0])

print("\nNegative #1:")
print(valid[0]['negatives'][0])

print("\nNegative #2:")
print(valid[0]['negatives'][1])

License and Terms of Use

License: The Dataset is made available under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

By downloading this Dataset, you agree to comply with the following terms of use:

• Restrictions: You agree not to use the Dataset in any way that is unlawful or would infringe upon the rights of others.
• Acknowledgment: By using the Dataset, you acknowledge that the Dataset may contain data derived from third-party sources, and you agree to abide by any additional terms and conditions that may apply to such third-party data.
• Fair Use Declaration: The Dataset may be used for research if it constitutes "fair use" under copyright laws within your jurisdiction. You are responsible for ensuring your use complies with applicable laws.

Derivatives must also include the terms of use above.

Citation

If you use our dataset, please cite our work as follows:

@misc{lu-2024-weblinx,
      title={WebLINX: Real-World Website Navigation with Multi-Turn Dialogue}, 
      author={Xing Han Lù and Zdeněk Kasner and Siva Reddy},
      year={2024},
      eprint={2402.05930},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}