UIS-QA: A Benchmark for Unindexed Information Seeking

Figure 1. UIS problem. Standard agents (bottom) rely on indexed information and often fail or hallucinate; UIS-capable agents (top) use additional tools to excavate unindexed information and solve UIS tasks.

_{If .figs do not load, see the paper.}

---

🔔 News

• [2026.03.10] 🎉 We release the UIS-QA dataset and the paper (ICLR 2026, arXiv) today!

---

📋 Dataset Description

Summary

UIS-QA is the first benchmark dedicated to Unindexed Information Seeking (UIS)—the setting where the information needed to answer a question is not directly retrievable via search engine results (e.g., content behind deep navigation, dynamic pages, embedded files, or overlooked corners of the web). It is introduced in the ICLR 2026 paper UIS-Digger.

Unlike conventional information-seeking benchmarks (e.g., GAIA, BrowseComp), UIS-QA explicitly requires agents to rely on unindexed information: answering correctly demands actions such as multi-step browsing, option selection, filter setting, file download, or reading content that search snippets do not expose. The benchmark is designed to evaluate whether agent systems can discover and use information that is not in standard search indices.

---

📊 Main Results (from the paper)

Evaluation results on UIS-QA, GAIA, and BrowseComp-zh (BC-zh). Action space: crawl (read webpage content), visual (read images), file (download/read files), browser (operate browser). ✓ = supported, ✗ = not supported.

† Reasoning-oriented LLMs. ‡ GAIA-text-103 (not full GAIA). § Memento without case bank (UIS is a new task).

Best on UIS-QA: UIS-Digger reaches 27.3% (tied best), outperforming all baselines including those with O3 or GPT-4.1.

---

🔧 UIS-Digger Framework and QA Construction Pipeline

Figure 2. UIS-Digger multi-agent system. Planner, web searcher, web surfer, and file reader work together. The web surfer can switch between textual and visual mode.

Figure 3. QA pairs construction pipeline. Left: Real-world information → homepage collection → unindexed information collection → question generation. Right: Simulated webpages with difficult action spaces → QA generation from JSON DB. (If the PDF does not render in your viewer, see the paper.)

---

📁 Dataset Structure

Data fields

Data format

• File: uis_qa.tsv (tab-separated values).
• Encoding: UTF-8. Fields may be wrapped in triple quotes (""") when they contain tabs or newlines.
• Splits: No predefined train/validation/test split; the dataset is a test set for benchmarking.

Example

Question	Answer
"""How many restrooms does the Metropolitan Museum of Art have, and how many of them are accessible restrooms?"""	"""13 restrooms, 10 accessible restrooms"""

(The dataset also contains Chinese questions; the file is UTF-8 encoded.)

Answer encryption (anti-leakage)

To reduce the risk of the benchmark being crawled or used for pretraining (so that models memorize answers instead of performing real information seeking), the Answer column may be released in encrypted form on Hugging Face. Answers then appear as UIS_ENC_V1: followed by ciphertext and are not usable for training.

Evaluators: To obtain the decryption key, please fill in the key request form (with reCAPTCHA verification). The key will be sent automatically to your email after submission. Then use the provided decoder script for local evaluation:

pip install cryptography
python scripts/decrypt_answers.py <encrypted.tsv> -o <decrypted.tsv> --key-file <key.txt>

See scripts/README.md for full instructions. Do not redistribute the key or decrypted answers beyond personal evaluation use.

---

📝 Data Collection and Curation

Problem formulation (from the paper)

• Indexed information (II): Information present in search result snippets or in one-step crawl from indexed pages.
• Unindexed information (UI): All other information on the web (deeper pages, files, dynamic content, etc.).
• A question is a UIS task if: (1) solving it requires some unindexed information, and (2) the correct answer cannot be inferred from indexed information alone.

Collection procedure

1. Expert annotation: An expert group manually created question-answer pairs by:
   • Navigating authoritative or official websites (government, companies, museums, repositories, etc.).
   • Performing interactive actions: multi-round clicks, option selection, filters, site-internal search, file download.
   • Reaching a specific information source (page or file), then formulating a question whose answer is in that content.
2. Diversity: At most two QA pairs per website to encourage coverage across domains and sites.
3. Filtering: A UIS filtering pipeline was applied to remove questions solvable using only indexed information (see below).

Curation principles

UIS filtering pipeline

To ensure each question truly requires unindexed information:

1. Manual check: Three annotators independently used Google Search to verify that the target content is not directly in the search result page (if the SERP only links to the content page, the question is still considered UIS).
2. Automatic verifier (z.ai): Used to filter out questions answerable from search alone; questions that require downloading a file to answer are kept as UIS.
3. LLM filter: An offline LLM (e.g., DeepSeek-R1) was used to remove questions answerable from the model’s internal knowledge alone.

The final set consists of 110 high-quality UIS samples.

---

🌐 Task and Domain Coverage

• Task type: Information seeking with final answer–oriented evaluation (short, deterministic answers).
• Environment: Real-world live web; unknown start point (agents begin from a general-purpose search engine).
• Required capabilities: Search, crawl, optional file download/parsing, and webpage interaction (click, scroll, select, type, etc.).

Domains (non-exhaustive): government announcements, official product/data pages, source code repositories, games (e.g., esports, game wikis), company reports, museums and cultural institutions, finance and markets, academic and patent databases, sports, and general factual lookups from official or authoritative sites.

---

✅ Evaluation

• Metric: Accuracy (exact or rule-based match of the model’s final answer to the golden answer).
• Verification: The paper uses a rule-based LLM as an automatic verifier; some answers include explicit criteria (e.g., “either 117 or 115 is correct”).
• Important: Evaluation should be performed with an agent that has access to search, browsing (and optionally file reading), not with a single-call LLM API, so that unindexed information can actually be sought.

---

⚠️ Limitations and Considerations

---

🎯 Intended Use

• Primary: Benchmarking information-seeking agents (with search + browsing ± file tools) on their ability to find and use unindexed web information.
• Research: Studying failure modes (e.g., retrieval vs. reasoning), action space design, and training strategies for UIS-capable agents.
• Not intended: Training data without proper licensing of underlying sources; evaluation of pure LLMs without tools; non–information-seeking tasks.

---

📎 Citation

If you use UIS-QA in your work, please cite the ICLR 2026 paper:

@inproceedings{uis-digger-iclr2026,
  title     = {UIS-Digger: Towards Comprehensive Research Agent Systems for Real-World Unindexed Information Seeking},
  author    = {Liu, Chang and Kuang, Chuqiao and Zhuang, Tianyi and Cheng, Yuxin and Zhou, Huichi and Li, Xiaoguang and Shang, Lifeng},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026}
}

---

📜 License

This dataset is released under the Apache-2.0 license.

---

🔗 Related Resources

• UIS-Digger: The multi-agent framework and baseline introduced in the same paper; achieves 27.27% on UIS-QA with a ~30B-parameter backbone and SFT + RFT training.
• Paper & code: See the paper for methodology, baselines, and analysis of failure modes and agent behavior.