mini-swe-agent asks how much work a capable model can do with a small agent around it. We wanted to explore that idea in a browser, with screenshots and the same controls you use to navigate a website.

A short implementation gives you room to experiment. Change the instructions, try another model, or add an action. You can follow the run and see how those changes affect it.

Once you have completed the setup below, install FFmpeg and record a run:

uv run demo/record.py --model google/gemini-3.8-flash

The agent works through a booking on a local page while the recorder captures the browser and logs its actions. It checks the final confirmation, then saves demo/demo.gif and demo/demo.json. FFmpeg is only needed to make the GIF.

To run the agent without cloning the source:

python -m pip install mini-web-agent
python -m playwright install chromium --no-shell

export OPENAI_API_KEY="your-openrouter-key"
export OPENAI_BASE_URL=https://openrouter.ai/api/v1

mini-web-agent --headed --cursor --model google/gemini-3.8-flash \
  'Open https://example.com and tell me the heading.'

The installed command uses the same CLI as python agent.py. You can also run python -u -m agent with the same options.

If you prefer to set up Python yourself, use Python 3.10+ and install the project this way:

git clone https://github.com/xhluca/mini-web-agent.git
cd mini-web-agent
python3 -m venv .venv
source .venv/bin/activate
python -m pip install .
python -m playwright install chromium --no-shell

export OPENAI_API_KEY="your-openrouter-key"
export OPENAI_BASE_URL=https://openrouter.ai/api/v1

python -u agent.py --headed --cursor --model google/gemini-3.8-flash \
  'Open https://example.com and tell me the heading.'

You will see the same browser window and terminal output as in the uv example.

As you try longer tasks, you can change the model, set a turn limit, or keep using a browser that is already open:

To run another task from the project directory, replace the text in quotes:

uv run agent.py --headed --cursor --model google/gemini-3.8-flash \
  'Open https://example.com and tell me the heading.'

To try another provider, set OPENAI_API_KEY and OPENAI_BASE_URL for its endpoint. The endpoint must support the Responses API, and the model needs image input and function calling.

If Chrome is still running from an earlier session, connect using the same profile:

uv run agent.py --connect --profile .chrome --model google/gemini-3.8-flash \
  'Tell me what is open in the current tab.'

You pick up the browser with its existing tabs and display mode. Adding --headed prints a warning and continues, since connecting cannot change how Chrome was launched. The CLI closes browsers it launches. When you use --connect, it leaves Chrome running.

When launching Chrome, the CLI replaces restored tabs with one blank tab and keeps profile data. Use --port 9222 for a fixed launch port. With --port 0, Chrome chooses a port for a new profile. Existing profiles reuse their recorded port. --connect reads the profile's address and cannot be combined with a nonzero --port. The CLI prints the CDP address after connecting.

On minimal Linux systems, Chromium may also need system dependencies. Install them with uv run playwright install-deps chromium.

The default instructions include the source of agent.py, so the model can read the functions it may call. Their signatures and docstrings also provide the tool definitions sent to the API.

The implementation has two dependencies: Playwright for browser control and the OpenAI SDK for model requests. Requests use an OpenAI-compatible Responses API; the quick start uses OpenRouter. Models need to support both images and function calls.

The first screenshot is sent as a user message. After each batch of actions, the last tool result includes a new screenshot and the current tab information. Earlier screenshots and reasoning stay in history, so you can follow what the model received at each turn. Appending to that history also lets providers cache the unchanged prefix. Longer tasks use more context, and cache hits depend on the provider.

The model has 24 actions to choose from. They are ordinary functions grouped in Actions, so you can read exactly what each one does:

Clicks and pointer movements use coordinates from the screenshot. The viewport defaults to 1280×800; pass w and h to WebAgent to change it. type_text types into the focused field, with 10 ms between characters. Tab indices come from the latest observation and can shift after closing a tab.

A failed action becomes feedback for the next turn. Each action returns one of:

{"state": "success", "output": ...}
{"state": "error", "output": "..."}

The model sees the error and can try another action. If it returns no tool calls, the loop asks it to choose one. Incomplete responses are discarded and retried before any of their actions execute.

Each model turn counts toward max_steps, including retries. At the limit, the loop stops with an unfinished-task message. A successful finish returns the model's final answer.

The model sees screenshots and tab information. It uses the browser controls in the action dictionary to interact with pages. It has no tools for reading the DOM, parsing accessibility trees, running generated code, managing uploads or downloads, or controlling the OS.

The action dictionary limits which functions the model can call. The browser can still visit websites and interact with them normally.

Chrome runs as a separate process, so you can leave it open between tasks. Starting Chrome, connecting to it, and closing it are separate operations:

Use port=9222 for a fixed port, or leave it at 0 to let the agent choose. After launch, agent.port holds the port Chrome is using. To return to this browser later, create another WebAgent with the same profile and call connect().

Playwright connects through Chrome's debugging interface, CDP. It finds the current WebSocket address over HTTP, then uses that connection to control the browser.

Start by changing what you tell the model or what you let it do. action_space and instructions are required, so each run uses the actions and prompt you supply. get_action_space() and get_instructions() provide the defaults used in the example.

The action dictionary determines which functions the model can call and which tool definitions are sent to the API. Add or remove a function there to change its options.

Callbacks let you watch an action before or after it happens. Each receives (step, action, result): the model-turn index, the action's name and arguments, and its result. Before execution, result is None. For example, show the cursor before an action and print the result afterward:

from functools import partial
from callbacks.cursor import show_cursor

callbacks = [
    dict(type="before", function=partial(show_cursor, agent)),
    dict(type="after", function=print),
]

The cursor.py overlay glides between targets, follows drags, and pulses on clicks. It respects reduced-motion preferences and lets clicks pass through to the page. The CLI loads it when you pass --cursor.

When the model needs to talk to you, on_message displays its message and on_reply collects your answer. They default to print and input; replace them to use your own UI.