When Do Observations Become Essential in Deep Search?

Observation masking provides a narrow but informative lens on when context management helps, when retrieval is the bottleneck, and when saturated models start losing crucial evidence.

📰 News

• [2026.09.22] 🏆 Our paper and results are now live on Papers with Code! Marked as SOTA performance with DeepSeek-V4-Flash and ranks 3rd overall on both the BrowseComp-Plus (with AgentIR-4B) and BrowseComp-ZH (with Serper API) leaderboards.
• [2026.08.30] 🎉 Our paper has been accepted to EMNLP 2026!

Regime Overview

Deep search agents accumulate long observation traces while iterating through reasoning, search, and browsing. We study observation masking as a simple context-management (CM) intervention and find that its effect is highly regime-dependent across model-retriever pairs.

Main Results

CM adds little when the retriever is the bottleneck, because the context contains too little answer-supporting evidence to rescue. It helps most in the middle regime, where useful signals are sparse and masking removes noise that the model cannot yet filter. Once the base model is saturated, aggressive masking can collapse performance by evicting crucial evidence during long trajectories.

Features

• Model Compatibility: seamless support for Qwen3.5 / Qwen3.6 Family, DeepSeek-V4-Flash, NVIDIA Nemotron 3, and GPT-OSS.
• Parallel Tool Calls: optional parallel browser/search execution for models that benefit from multi-tool-call planning according to their native tool-calling template (Qwen3, DeepSeek-V4-Flash, NVIDIA Nemotron 3, and GPT-OSS).
• Configurable Masking Window: freely choose the window size for retaining or archiving stale observations.
• Flexible Search Backend: use local BM25, Qwen3-Embedding-8B, or AgentIR retrieval for BrowseComp-Plus, and Serper-backed web search for live-web benchmarks.
• Trajectory Preservation: keep full saved trajectories for analysis even when observations are masked or archived in the active model context.

🛠 Recommended Environment

• More/Better than 8 * A100 80G Nvidia GPUs on one node
• Linux Ubuntu 22.04.4
• CUDA 12.2 or higher (if you need to run Qwen3.5/3.6 and DeepSeek-V4, you will need CUDA 13.1 or higher)

One-click setup

One-click setup (optional, you can also install dependencies manually):

bash setup.sh

Manual Installation

# Optional: system-wide OpenJDK 21, only if you have sudo.
sudo apt update 
sudo apt install -y openjdk-21-jdk

# No sudo? Skip the apt commands above.
# setup.sh will download OpenJDK 21 into ./.jdk/jdk-21 when needed.

# install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.12
source .venv/bin/activate

# install tevatron for BrowseComp-plus 
git clone https://github.com/texttron/tevatron.git
cd tevatron
uv pip install -e .
cd ..

# install all dependencies automatically
uv pip install -e .

Configure API keys for Serper-backed benchmarks and evaluation:

cp .env.template .env

Remember to config your API keys in .env file:

# Get API Key: https://serper.dev/
SERPER_API_KEY=your_serper_api_key_here

# OpenAI API for evaluation: https://platform.openai.com/api-keys
OPENAI_API_KEY=your_openai_api_key_here

Deploy Search

BrowseComp-Plus uses a local search service on custom ports.

Attention: The dense and AgentIR search service will occupy about 10G of your GPU.

# BM25
bash scripts/start_search_service.sh bm25 8005

# Dense (Qwen3-Embedding-8B)
bash scripts/start_search_service.sh dense 8006

# AgentIR
bash scripts/start_search_service.sh agentir 8007

Deploy Models

GPT-OSS-20B

bash scripts/start_gptoss_servers.sh \
  8010 \
  "2,3,4,5,6,7" \
  "openai/gpt-oss-20b" \
  "0.9,0.9,0.9" \
  2 \
  24

This starts 3 two-GPU TP replicas on ports 8010 through 8012.

Qwen3.5-9B (CUDA ≥ 13 required)

uv venv qwen35 --python 3.12
uv pip install vllm \
  --torch-backend=auto \
  --extra-index-url https://wheels.vllm.ai/nightly \
  --python qwen35/bin/python

bash scripts/start_qwen_servers.sh \
  8010 \
  "1,2,3,4,5,6,7" \
  "Qwen/Qwen3.5-9B" \
  "0.95,0.95,0.95,0.95,0.95,0.95,0.95" \
  1 \
  32

This starts 7 single-GPU replicas on ports 8010 through 8016.

For Qwen3.5/3.6-35B, GPT-OSS-120B, DeepSeek-V4-Flash, and NVIDIA Nemotron 3 deployment recipes, see assets/docs/deployment.md.

Run

The last run_agent.sh argument controls max_concurrency_per_worker. Recommended values are 8 for DeepSeek-V4-Flash (dsv4) and GPT-OSS-120B, 24 for GPT-OSS-20B, and 32 for other models.

BrowseComp-Plus with AgentIR and GPT-OSS-20B

Start AgentIR search on port 8003, then run GPT-OSS-20B against BrowseComp-Plus:

bash scripts/start_search_service.sh agentir 8003

SEARCH_URL="http://localhost:8003" bash run_agent.sh \
  results/browsecomp-plus/gptoss-20b-agentir \
  8010 \
  3 \
  browsecomp_plus \
  local \
  "openai/gpt-oss-20b" \
  10000 \
  "" \
  on \
  24

BrowseComp-ZH with Serper, Observation Masking and Qwen3.5-9B

Serper-backed benchmarks do not need a local search service. Make sure SERPER_API_KEY is set in .env.

bash run_agent.sh \
  results/browsecomp-zh/qwen3.5-9b-serper \
  8010 \
  7 \
  browsecomp-zh \
  serper \
  "Qwen/Qwen3.5-9B" \
  4 \
  "" \
  on \
  32

Evaluation

python eval.py --input_dir results/browsecomp-plus/gptoss-20b-agentir --model_name_or_path openai/gpt-oss-20b

python eval.py --input_dir results/browsecomp-zh/qwen3.5-9b-serper --model_name_or_path Qwen/Qwen3.5-9B

For full script arguments, see assets/docs/parameter.md.

Scaffold

We decouples browser execution from model serving: a shared browser pool handles search, page opening, and observation collection, while model workers consume the resulting trajectories through configurable masking windows. This makes it easy to compare model-retriever pairs under the same browsing environment and CM policy.

Benchmarks

For benchmark notes, see assets/docs/benchmarks.md.

Analysis & Findings

Sparse Signal and Complex Inputs

Observation masking (CM) helps most when the useful signal is sparse and the input trace is complex. Each point represents a sampled No-CM input prefix: the x-axis is the first principal component over input-trace features, where larger values indicate greater complexity, and the y-axis is the normalized fitted SNR. Green points are CM-rescued cases, while red points are unchanged cases. Saturated models show more separable rescued subsets, whereas retriever bottlenecks weaken the baseline signal and sharply suppress that separability.

Reasoning Attention over Tool Results

Observation masking improves context management when it preserves the tool results that the model still uses for reasoning. The analysis measures reasoning-token attention over tool-result tokens: cases with stronger CM gains maintain higher reasoning attention on useful tool results, while weaker settings show lower or more diffuse attention. The three model-retriever settings are Qwen3.5-4B + BM25, Qwen3.5-9B + AgentIR, and Qwen3.6-35B-A3B + AgentIR.

Page Reopening -- Lost in the Middle

Agents reopen middle pages much less often. The figure shows the relative positions of open targets in the current page pool, where CM sharpens the U-shaped pattern: agents tend to revisit early or recent pages more often than pages in the middle of the pool.

i-DeepSearch Team

We are grateful for all the help we got from our contributors.

Acknowledgements

           

We also thank the following open-source projects:

• vLLM for fast LLM inference support.
• OpenResearcher for early-stage exploration in scaffold-building and deep search benchmark collection.
• DeepSeek-AI, Qwen-AI, and OpenAI GPT-OSS Team for their marvelous open-weight models.

Citation

@article{zhang2026masking,
  title={Masking Stale Observations Helps Search Agents -- Until It Doesn’t: A Regime Map and Its Mechanism},
  author={Zhang, Haoxiang and Xu, Qixin and Li, Zhuofeng and Zhang, Lei and Jiang, Pengcheng and Zhang, Yu and McAuley, Julian},
  journal={arXiv preprint arXiv:2606.00408},
  year={2026}
}