GeoGuesser

A GeoGuessr-style visual geolocation environment. The agent is dropped at an unknown street-level location, looks around, walks along the road, pins candidate coordinates on a map to check itself, and commits to a final guess. Reward is distance-based, using the game's own scoring curve.

Independent open-source project, unaffiliated with GeoGuessr AB. Imagery comes from Mapillary contributors under CC-BY-SA-4.0.

Quick start

cd envs/geoguesser_env

# The frozen 200-task eval split is committed, so this runs as-is, with the
# same configuration the Space uses.
./scripts/serve_local.sh             # http://localhost:8000/web/

# To build your own data (needs a free Mapillary token with READ scope):
export MAPILLARY_API_KEY_TRAIN="MLY|..."
python scripts/harvest_tiles.py                        # enumerate sequences
./scripts/build_dataset.sh                             # mirror tasks offline
python scripts/verify_offline.py tasks/pool_offline_5k.jsonl
python scripts/split_tasks.py tasks/pool_offline_5k.jsonl --eval 200

from geoguesser_env import GeoGuesserEnv, GuessAction, LookAction, PinAction

env = GeoGuesserEnv(base_url="http://localhost:8000")

result = env.reset(split="eval", index=7)     # byte-identical on repeat
print(result.observation.prompt)

result = env.step(LookAction(heading_deg=90, fov_deg=45))
result = env.step(PinAction(lat=-16.5, lon=-68.1))
print(result.observation.feedback)
# Pin 1 placed at -16.5000, -68.1000 - Bolivia (South America).
# Nearest major city: La Paz, ~5 km E. 10 actions left.

result = env.step(GuessAction(response="Altiplano. <guess>-16.49, -68.12</guess>"))
print(result.reward, result.observation.distance_km)

Tools

Tools the backend cannot serve are not registered, so the agent never sees a tool that always fails.

The agent's map and the player's map agree

The player sees live OpenFreeMap tiles; the agent sees an offline Natural Earth render. They have to agree about how precisely a pin can be aimed, because that is what the distance reward measures — a map showing only country outlines lets you place a country, not a point within a city.

So the guess map is zoom-aware. place_pin takes span_deg, and the render adds detail as the window tightens:

Natural Earth tops out at highway level — it shows the motorways around a city but not the grid inside it. Below 0.35 degrees the map therefore fetches actual ways from Overpass, generalising by zoom the way a real style does: minor classes appear only once the window is tight enough to hold them, and widths grow as it shrinks. A pin on Abuja at span_deg=0.05 came back as an 11 km window with the full street grid, drawn white-on-pale to read like the player's Positron tiles.

Overpass has real limits, and they are the binding constraint on how this scales: roughly 10,000 requests and 1 GB per day, about 2 concurrent slots per IP, a 180 s runtime and 512 MiB memory ceiling per query, HTTP 429 when rate limited and 504 when a query is too large. Cooldowns lengthen for heavy users. So street detail is right for eval, demos and modest training, and the cache is what keeps it polite — a run doing millions of pins must pre-warm or bundle a Protomaps extract instead.

The first render of a neighbourhood costs 3-16 s; every later one is served from data/geo/osm_cache/ in ~30 ms and is byte-identical. That makes an episode deterministic once warm, and a frozen eval should pre-warm the cache the same way it pre-warms panoramas — or set GEOGUESSER_STREET_DETAIL=0, which falls back to Natural Earth and never touches the network. Any fetch failure degrades to no streets rather than failing the step.

The optional detail layers are fetched once, since 87 MB of GeoJSON does not belong in the repo:

python scripts/fetch_detail_geo.py     # compacts to ~39 MB, gitignored

Without them the map still renders, with outlines and major cities only.

In the play page the pin carries the zoom you are actually looking at, so the "what the agent sees" panel is framed like your own view at the same scale and with comparable detail.

Overpass has a usage policy that discourages heavy automated querying, so this is right for eval, demos and modest training, and the cache is what keeps it polite. A run doing millions of pins should pre-warm or bundle a Protomaps extract instead.

Zoom resolves real detail

Zooming is not cosmetic, but it needs the right source. A 30-degree view of a 2048x1024 panorama samples only about 170 source pixels, so narrowing the field of view barely adds information — measured mean gradient 6.60 at 90 degrees against 7.03 at 30. The 7680x3840 original roughly doubles it (10.23 against 14.87), which is the difference between guessing at a sign and reading it.

So each panorama is cached twice. Wide views render from the 2048 derivative in ~30 ms; a field of view at or below 45 degrees pulls the original and renders in ~70 ms. If no original exists the step degrades to a soft view rather than failing. Set GEOGUESSER_HIRES_ZOOM=0 to disable it.

Pinning tells you where you pointed, not whether you are right

place_pin returns a rendered map and a description of the pinned location: country, subregion, nearest city with distance and bearing, and the distance to the agent's own earlier pins. It reveals nothing about the target.

That restraint is deliberate. Any signal about the truth — a distance, a warmer/colder hint — would make binary search the optimal policy, and the environment would measure bisection rather than geographic reasoning. Distance and score arrive only from submit_guess.

Reward

geo     = exp(-distance_km / 1492.7)           # GeoGuessr's curve, in [0, 1]
partial = 0.15 * country_hit + 0.10 * region_hit   # when hierarchical
cost    = 0.01*looks + 0.01*maps + 0.02*pins + 0.05*moves
reward  = clip(geo + partial, 0, 1) - cost

An unparseable or out-of-range guess scores 0.0 and says why. Parsing accepts what models actually emit: decimal pairs, DMS (48°51'29"N), labelled lat:/lon:, JSON, and <guess> tags.

Training against it

The defaults above reproduce the game, which makes a score directly comparable to GeoGuessr. They are the wrong shape for RL, and three flags change that:

GEOGUESSER_REWARD_SHAPE=mixture \
GEOGUESSER_COST_MODE=multiply \
GEOGUESSER_HIDE_IDENTITY=1 \
    uvicorn geoguesser_env.server.app:app

Replaying all 3,037 recorded eval episodes through both settings: episodes scoring exactly zero fall from 10-46% to 0% for every model, and the leaderboard order only changes within the tiers already documented as inside noise at n=200.

The terminal observation carries full provenance either way — once the truth is revealed it can no longer be used to shortcut the episode — so recorded traces stay complete under hide_task_identity.

Baselines

Measured with examples/geoguesser_llm_rollout.py on the committed index, tasks 0/7/14/21/28, so the numbers are reproducible rather than illustrative. Five episodes is far too few for a leaderboard; they are a smoke test that the task is solvable and the reward is discriminative.

Sonnet placed two guesses within 2 km. The agentic score sits slightly below single-shot on the same tasks because looking around costs reward and the extra views did not always pay for themselves — which is the trade-off the environment is meant to expose, not a defect.

Qwen does follow the multi-turn protocol: across the agentic runs it produced look, move, zoom, pin and guess actions and navigated up to 68 m down a road. Two things had to be right first, and both are prompting or plumbing rather than capability:

• Token budget. Reasoning models put their chain of thought in a separate reasoning_content field and can exhaust the budget before emitting any content, which looks exactly like a model that cannot see images. At 1,024 tokens Qwen scored 0/5 with empty replies; at 3,500 it followed the protocol intermittently, failing turns whose reply came back as pure reasoning; at 8,000 it parsed 4/4 turns. The example defaults to 3,000 and takes --max-tokens.
• A deadline. Left to itself Qwen explored until the turn budget ran out and scored zero. The agentic prompt now warns it explicitly when two turns remain, after which it committed.

What remains is accuracy, not plumbing: its guesses landed 452 km, 579 km and 2,423 km out against Sonnet's 98 km median. It is also 10x slower — 110-193 s per agentic episode against Sonnet's 10-16 s.

python examples/geoguesser_llm_rollout.py --provider anthropic \
    --model claude-sonnet-5 --episodes 5
python examples/geoguesser_llm_rollout.py --provider hf \
    --model "Qwen/Qwen3.5-9B:together" --episodes 5 --max-tokens 4000
python examples/geoguesser_llm_rollout.py --provider anthropic \
    --mode agentic --episodes 3 --verbose

Parallel rollouts

An episode is stateful, so concurrent rollouts each need their own environment instance over the shared read-only index and cache. --concurrency does that:

python examples/geoguesser_llm_rollout.py --provider hf \
    --model "Qwen/Qwen3.5-9B:together" --episodes 8 --concurrency 4

Eight Qwen episodes took 113.6 s wall against 334.0 s of summed latency — a 2.94x speedup on 4 workers, the shortfall being the provider's own queuing rather than the environment, which spends ~28 ms on a reset.

Scaling: what the environment can actually supply

Measured on an 18-core machine with a warm cache and street detail off, one environment per worker, with a correctness assertion in the loop so an interference bug cannot masquerade as throughput.

Per-step cost is dominated by map rendering, not imagery:

That shapes the two configurations:

Two facts fall out of that. look releases the GIL — the reprojection is numpy and the encode is Pillow — so threads scale well for view-only work and processes only add startup cost. Map rendering is pure Python, so it is GIL-bound and needs processes, which buy about 1.8x before contention.

GEOGUESSER_REVEAL_MAP=0 is the single biggest throughput lever: a training-shaped episode drops from 389 ms to 117 ms, since the reveal map is 280 ms that a training run never reads — the reward and the distance are in the observation either way. Keep it on for evals, traces and the UI.

Memory is small: about 215 MB for the imports, 100 MB more once the geodata caches fill, and ~13 MB per additional environment in the same process. The Natural Earth layers are process-shared through an LRU cache, so threads are far cheaper than processes here too.

Rate limits that actually bind

With a warm cache the environment makes no network calls at all. Default pins use a 7 degree span, which is above the street threshold, so they do not touch Overpass either — only a deliberately zoomed pin does. For an eval sweep set GEOGUESSER_STREET_DETAIL=0 unless you have pre-warmed, since 100 zoomed pins against 2 concurrent slots would throttle immediately.

100-task eval

Sonnet agentic measured 13.1 s per episode, so 100 episodes cost about 1,310 model-seconds and the concurrency you can use is set by the provider, not by this environment:

• concurrency 4 → ~5.5 minutes
• concurrency 8 → ~2.7 minutes, if your token-per-minute ceiling allows it

An agentic episode sends roughly five 640x640 views, about 550 tokens each, plus a growing text prompt. At a 400k input-tokens-per-minute ceiling that is roughly 6 episodes in flight before tokens, not latency, become the limit — which is why 4 to 6 is the practical range for Sonnet. Qwen through the router reached 3.97x on 6 workers, so 6 to 8 there.

Meanwhile the environment can supply 3.5 episodes/s in eval configuration against the 0.3 to 0.6 episodes/s those concurrencies actually consume, so it has roughly ten times the headroom it needs.

1000 training steps

For GRPO with 8 prompts and a group of 16, that is 128 episodes per step and 128,000 episodes, or about 640,000 env steps:

• environment time: 128,000 / 31.7 ≈ 1.1 hours total, spread across workers
• imagery: nothing, with a warm cache
• storage: 100 tasks warm is 60 MB; 5,000 tasks would be about 1.5 GB of start frames

So the environment is not the constraint — 640,000 model calls are, which needs batched local inference rather than an API. The constraint that is ours is task diversity: 128,000 episodes over 100 tasks means each location is seen 1,280 times, which is memorisation territory. Before a run that long, either harvest more tasks or add seeded heading augmentation, which multiplies effective tasks 8 to 12 times from imagery already on disk.

Readiness audit

scripts/readiness_check.py checks the properties that only appear at real scale and concurrency, rather than on the four committed fixtures:

python scripts/readiness_check.py --full

[PASS] index integrity            100 tasks, 47 countries, 100 unique sequences
[PASS] all tasks render           100/100 rendered, median reset 28 ms
[PASS] cross-process determinism  two subprocesses and this process agree
[PASS] parallel isolation         8 concurrent episodes, each its own task
[PASS] offline with warm cache    12/12 served with fetching disabled
[PASS] reward is discriminative   uniform-random 0.029, fixed-point 0.111
[PASS] step latency               look 29 ms, pin + map 249 ms
[PASS] one guess per episode      a second guess returns reward=None
[PASS] pin never leaks the target 60 pins across 20 tasks revealed nothing

The reward check matters most: a uniform-random guesser scores 0.029 and the best trivial constant guess 0.111, against Sonnet's 0.896. The signal is measuring geolocation rather than rewarding noise.

Tracing a rollout

--trace-dir records every turn: the image the model saw, what it said, the action it chose, the environment's reply, steps left and running cost. A second script renders that as one self-contained HTML page, which is the difference between knowing the reward and seeing why:

python examples/geoguesser_llm_rollout.py --provider anthropic \
    --mode agentic --episodes 6 --trace-dir rollouts
python scripts/render_trace.py rollouts/anthropic_agentic

Images are written beside the trace rather than inlined, since six agentic episodes carry around 35 views and a JSONL with those in it is neither readable nor loadable.

Every step carries an image, including the guess: a guess returns a reveal map with the guess, the true location and the line between them. Truth is drawn only there, after scoring. When the guess is more than 25 degrees out the second panel frames the true location instead of both points, because squashing a hemisphere into a panel shows nothing.

On the Hub

The Space and the bucket deliberately live in different namespaces: moving the Space should not mean re-uploading 22 GB, so deploy_hub.py takes the owner of each separately (GEOGUESSER_HF_ORG and GEOGUESSER_HF_BUCKET_OWNER).

The same client drives either:

env = GeoGuesserEnv(base_url="http://localhost:8000")                      # local
env = GeoGuesserEnv(base_url="https://huggingenvs-geoguesser-env.hf.space") # Space

Only the file locations differ — locally the indexes and imagery are in the repo, on the Space they arrive through the bucket mount. Splits, step budget, street labels and offline enforcement are identical, and verified so: the same task returns the same image checksums, reward and distance from both.

Deploy with scripts/deploy_hub.py --all. openenv push is deliberately not used: it cannot attach a bucket volume, and its default excludes would upload 22 GB of panoramas into git.

Local and Space parity

Verified across a local server and both Spaces, on both splits, driven by the same client. Every field below is identical in all three:

The guess map is pixel-identical too: mean absolute difference 0.000/255 over 1020x390. Only the PNG encoding differs, not the content.

The one fragile part is the street layer, which comes from Overpass. Overpass answers a laptop in ~2 s but intermittently returns 504 Gateway Timeout to datacenter egress, which once left the Space rendering coarse Natural-Earth-only maps while a laptop drew the full labelled grid. A single retry recovers it, and the environment now reports the state rather than degrading silently: observation metadata carries street_detail as on, off or unavailable, so a poorer map is visible instead of looking like a styling choice. Scoring is unaffected either way -- reward is distance-based. GEOGUESSER_OSM_CACHE points the street cache at a mounted, pre-warmed directory for deployments that cannot reach Overpass at all.

Task API

The environment implements the core TaskProvider protocol, so the Task API routes core already registers become live. Task discovery is metadata only; it never starts an episode.

curl localhost:8000/geoguesser_env/splits
# [{"name":"train","type":"train","num_tasks":3452,"default":true},
#  {"name":"eval","type":"test","num_tasks":200,"default":false}]

curl -X POST localhost:8000/geoguesser_env/num_tasks -d '{"split":"eval"}'
curl -X POST localhost:8000/geoguesser_env/task      -d '{"split":"eval","index":12}'

env.list_splits()          # [{"name": "eval", "type": "test", ...}, ...]
env.num_tasks("eval")      # 200
env.get_task("eval", 12)   # metadata, no coordinates and no country

Task specs are deliberately truth-free. They carry task_index, task_id, split, n_frames, provider, sequence_id and offline_ready — never coordinates and never the country. A spec travels to whatever orchestrates a run, and a label sitting in a spec can reach a prompt. The true location is revealed in observation metadata after the guess, which is the one place it belongs. Per-country eval breakdowns therefore come from finished episodes, not from list_tasks.

Splits are invisible to the agent. RESERVED_TOOL_NAMES blocks a reset MCP tool, so there is no way for a policy to see or choose its own task — the "agents cannot reset" invariant.

Collecting evals

scripts/collect_eval.py runs models against a split and records everything about each episode. One JSONL line per episode, ~13 KB.

export ANTHROPIC_API_KEY=...
python scripts/collect_eval.py --provider anthropic --model claude-sonnet-5 \
    --split eval --max-turns 12

# several endpoints in one run, each at its own concurrency
cp models.example.json models.json     # anthropic / openai / HF router / vLLM
python scripts/collect_eval.py --models models.json --split eval

Output lands in rollouts/<run_id>/: episodes.jsonl plus a run.json with per-model aggregates (mean reward, median distance, within-1/25/200/750 km, parse rate, forced guesses, tokens, latency). Runs are resumable — re-run the same --run-id and it skips episodes already recorded.

Each turn carries the camera state before and after the action, which is what makes a rollout re-renderable: a pan from 0 to 270 degrees can only be animated if both ends are known. It also carries the exact prompt, the raw reply, separated reasoning, finish reason, token counts, per-attempt errors and latency, and the provider's own response id.

Pixels are not stored. The environment is deterministic and the panoramas are local, so a renderer replays the state trajectory instead. Storing them would cost roughly 14 GB for 200 tasks across six models, all re-derivable. What is stored is a sha256 per observation, so a replay can be verified rather than assumed:

python scripts/verify_replay.py ../../rollouts/<run_id>/episodes.jsonl
# 3 episodes · 19 view turns checked · 19 match · 0 mismatch
# every view turn reproduces byte-for-byte

That check is not decorative: it exits non-zero on drift, because a video built from a mismatched trace looks authoritative and shows something the model never saw. Guess maps are excluded from it by construction — they depend on the Overpass response of the moment. Use --save-frames when you want the literal bytes anyway.

Rollout video

Two steps, split where the work naturally divides. Python owns pixels-from-panoramas, because the gnomonic reprojection already lives here and is verified byte-exact against the trace. React owns layout, typography and transitions, because that is where iterating on them is pleasant.

# 1. Frames + timeline.json, straight into the Remotion project's public/
python scripts/render_rollout.py ../../rollouts/<run>/episodes.jsonl --episode 0

# 2. Compose
cd video && npm install
npx remotion render Rollout out.mp4 --props=public/rollouts/<slug>/timeline.json
npx remotion studio          # iterate on the composition live

The pan is a real pan. A look from 0 to 270 degrees is not a cut between two stills: it renders one intermediate gnomonic reprojection per video frame along the shortest angular path, eased like a camera rather than swept linearly — 350 to 10 degrees pans +20, not -340. Zoom interpolates the field of view the same way. Verified: a look segment produces 29 distinct images, a zoom 24, with no duplicates.

Layout is a large panorama viewport with a HUD of the state the agent is acting on (heading, fov, actions left, cost), and a trace pane that reveals turns as they happen — the active turn carries the model's raw reply, past turns recede. Then a score card with the guess against the truth.

Fidelity is checked before anything is composed: every view keyframe is re-rendered at the size the model saw and compared to the trace's sha256, and a mismatch aborts. A video that looks authoritative while showing something the model never saw is worse than no video.

Reproducibility

env.reset(split="eval", index=7)  # exact task, byte-identical    -> GRPO, eval
env.reset(split="train", seed=42) # tasks[42 % n_tasks]           -> replay
env.reset()                       # random task in the default split, split and
                                  # index both recorded in metadata -> UI

task_index= still works as an alias for index=, so trajectories recorded before splits existed still replay. The split is recorded in observation metadata: without it a bare index is ambiguous across three indexes, and a trajectory stops being replayable.

Byte-identical repeats hold because panorama bytes come from a local cache rather than an expiring CDN URL, reprojection is pure numpy with integer sampling, and the initial heading is pinned to each panorama's own compass_angle.

An eval score is only meaningful alongside its provenance — the env version, the task index, and GEODATA_VERSION from server/render/minimap.py, since the bundled vectors determine the reverse-geocode text the agent sees.

Training and collection

The environment plugs into openenv.core.harness, so a rollout function and a collector come for free:

from geoguesser_env import GeoGuesserEnv
from geoguesser_env.harness import GeoGuesserSessionFactory, load_tasks

tasks = load_tasks("tasks/train_pano_v3.jsonl", repeat=16, split="train")
factory = GeoGuesserSessionFactory(
    lambda: GeoGuesserEnv(base_url="http://localhost:8000")
)

See examples/geoguesser_rollout.py for a scripted rollout and examples/geoguesser_collect.py for JSONL collection with resume.

Human play

A five-round game, 5,000 points a round on the same curve the environment rewards, so a human score is directly comparable to GeoGuessr intuition and to the agent's reward (both are shown).

A round is one episode with one guess. The five-round game is a UI wrapper around five separate episodes; the environment itself never accepts more than one guess, because submit_guess is terminal.

The page plays through the environment rather than simulating it. It opens the same WebSocket session API a client uses, calls reset(task_index=...), and sends every pin, look, zoom and move as a real charged step — so the step counter, the accumulated cost and the final reward are the environment's own numbers, not the browser's. A side panel shows the observation stream an agent would receive, including the environment's own rendered map and views.

Note that plain REST /step builds a fresh environment per request, so a stateful episode has to run over /ws; the Python client does this already.

uv run --project . server
# then open http://localhost:8000/geoguesser/play

The page stands alone at /geoguesser/play and is also embedded in the Gradio playground's Custom tab when the web interface is enabled:

ENABLE_WEB_INTERFACE=true uv run --project . server   # http://localhost:8000/web/

Pick a split and an episode with the reset(split=) and reset(index=) controls above the game and press load episode, or random episode — the same call an eval harness makes, so you can replay exactly the episode an agent saw. Those controls live on the Gradio side because choosing a task is orchestration, not something the player does mid-round; the page itself reads ?task= from its URL, so /geoguesser/play?task=42 opens that episode directly.

Drag to look around and scroll to zoom (free, for orientation). The look() and zoom(30) buttons run charged environment steps and show what the agent sees. Arrows, or the arrow keys, walk the road — the main view follows, keeping your heading. M toggles a larger map, T the trace panel, Enter submits and then advances. On submit the map takes the screen and draws the line between guess and truth, exactly like the game; the result bar shows distance, points, env reward and the true location, and a scoreboard breaks down all five rounds at the end.

Panoramas are rendered by Pannellum and the map by MapLibre over OpenFreeMap tiles — no API key, no request limits. The imagery credit line names the Mapillary contributor, which the CC-BY-SA licence requires.

The page has to be a standalone document rather than a Gradio gr.HTML fragment: gr.HTML inserts markup without executing <script> tags, so the viewers never initialise and the panel renders blank with no error anywhere.

Extra routes, all local:

The human map uses live tiles; the agent's map stays the offline Natural Earth render, so the agent keeps a determinism the browser does not need. Note that /geoguesser/task/{i} exposes ground truth — it exists for a person playing in their own browser, and agent observations still withhold it until the guess.

Configuration

Data

Three splits, carved from one 3,673-task pool so contamination is enforced exactly once, at split time, rather than reasoned about across two harvests:

Separation follows the OSV-5M rule: no shared sequence_id, and no training task within 1 km of an eval task. Frames sit ~3.3 m apart, so holding out an image while keeping its neighbour holds out nothing. The split script verifies its own work and exits non-zero if either rule is violated — the committed split reports 0 shared sequences and a closest train task 1.07 km away.

eval is committed because a frozen benchmark belongs in version control, where a change to it shows up in review. The training index and the imagery live in a Storage Bucket, mounted read-only at /data on a Space.

The random split is not yet implemented — the plumbing takes arbitrary named splits, but the pool-backed sampler is still to come.

Each index is self-contained: every frame's coordinates, heading and capture date live in the JSONL, so the movement graph resolves offline. Only image bytes are fetched, and only on a cache miss, because Mapillary thumb_*_url values are expiring signed URLs that cannot be stored.

Coverage is uneven and worth knowing about. Probing 45 Street-View coordinates found any Mapillary imagery at 21 and a 360-degree panorama at only 7, heavily clustered. Panorama-first discovery is therefore the only approach that works — roughly 5% of probe points yield a usable sequence, so reaching 100 tasks took two passes with different seeds, merged by scripts/merge_task_indexes.py. Africa and Oceania are thin because 360-degree contributors are; that is a property of the source, documented rather than papered over.

Known gaps versus the real game

Movement follows captured sequences and stops where one ends. There is no multi-round cumulative score, no wall-clock timer (a step budget stands in for it), and no satellite layer on the guess map. Coverage hints and web search are deliberately excluded: the first is a crutch, the second turns the task into retrieval.

See DESIGN.md for the reasoning behind these choices.