greynet — Competitive Scraping Challenge

A gamified, submit-the-data-not-the-code scraping competition. Participants scrape deliberately-defended practice targets; a backend validation engine scores the data they extract (completeness, speed, stealth) on a live leaderboard. Runs entirely on free tiers (Vercel Hobby + Upstash Redis).

Part of the talk "Navigating the Grey: Scaling from Single Worker to Multi-VM Undetectable Scrapers." All data is synthetic; targets are practice sandboxes.

Three target sites of increasing difficulty, all built, each forcing a specific taught technique. Full organizer reference (endpoints, scoring, anti-cheat) in docs/challenges.md.

01 · OmniCorp Job Portal /megacorp — ENTRY

02 · The Syndicate /syndicate — ELEVATED

03 · Anomalous Specimen /specimen — SEVERE

Difficulty is cumulative — each site layers new defenses on the last.

Local development

npm install
cp .env.example .env.local      # all blank works — falls back to in-memory Redis
# regenerate challenge assets only if data/ + public/challenges are missing:
npm run gen:megacorp            # listings + OCR salary images
npm run gen:syndicate           # members + dossier PDFs + intercept audio (needs espeak-ng + ffmpeg)
npm run gen:specimen            # specimens + messy charts
npm run dev

No Upstash credentials? The app uses an in-memory Redis mock so everything runs offline. Geo-gating is inert locally unless you set GEO_SIMULATE_COUNTRY or ALLOW_GEO_DEBUG=1 (then send an x-debug-country header).

Verify it works

npm test                       # scoring/dot-path, mousegate, fingerprint, audio-match, validators (48 tests)
node scripts/proof-assets.mjs  # proves Tesseract reads the chart + dossier PDF
node scripts/proof-audio.mjs   # proves Whisper transcribes the intercept audio
node scripts/screenshot.mjs    # captures every page (chromium) to /tmp/shots
node scripts/run-solver.mjs <megacorp|syndicate|specimen>   # full end-to-end reference solver

Reference solvers live in scripts/solver-example/ and are runnable worked examples: register → beat each site's defenses (session/clearance/fingerprint) → extract via the hidden APIs + OCR (+ STT for the audio gate) → submit. The megacorp solver reaches ~99%; run-solver.mjs spawns the server and runs them.

Deploying to Vercel (free)

1. Create an Upstash Redis database (free tier) → copy UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN.
2. Import the repo into Vercel (Hobby). Set env vars:
   • UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN
   • TOKEN_SECRET (openssl rand -base64 48)
   • GEO_ALLOWED_COUNTRIES — set to a country your participants are NOT in (default US) so the restricted region forces a proxy.
   • Do not set ALLOW_GEO_DEBUG in production.
3. Commit the generated assets (data/megacorp.json, public/challenges/...). They're deterministic — regenerate any time with npm run gen:megacorp.

API route durations are capped at maxDuration = 60 to fit Hobby limits; all Redis counters have TTLs and timing logs are capped.

Design system

Two deliberately-clashing aesthetics reinforce the fiction (a hacker breaching a sterile corporate site). Carry these forward to Sites 2 & 3 — give each target its own distinct identity while keeping the greynet shell consistent.

• greynet platform — terminal noir / grey-market console. Near-black (--color-void) with an acid-phosphor signal accent (--color-signal), Chakra Petch display + JetBrains Mono, grain + scanline + glow textures, staggered rise reveals, blinking caret. Data-dense, technical.
• OmniCorp target — clinical dystopian megacorp (Severance/Lumon energy). Cold bone paper (--color-paper) + institutional navy/teal (--color-omni-ink / --color-omni-teal), Fraunces high-contrast serif + Hanken Grotesk, hairline rules, official-document framing (Form OC-1, classification strip).
• The Syndicate target — typewriter noir dossier. Warm near-black (--color-noir) + oxblood (--color-blood), Special Elite typewriter + JetBrains Mono, CONFIDENTIAL stamps, redaction tone.
• Specimen target — containment vault. Near-black (--color-vault) with hazard-amber (--color-hazard) + clinical-cyan (--color-clinical), hazard stripes, threat bars, Chakra Petch + JetBrains Mono.

Design tokens (colors, fonts) live in app/globals.css under @theme; fonts are wired in app/layout.tsx via next/font. Reusable texture/animation utilities: .grain, .scanlines, .glow-signal, .box-glow, .caret, .rise, .paper-grain.

Visual verification

Chrome isn't required — scripts/screenshot.mjs uses playwright-core with the system chromium (/usr/bin/chromium-browser). It self-spawns next start, captures every page to /tmp/shots, and tears the server down:

node scripts/screenshot.mjs

Architecture

• Next.js (App Router, TS, Tailwind) — route groups split the platform (app/(platform)) from the target sites (app/(targets)).
• Upstash Redis — teams, tokens, rate-limit/breaker counters, timing logs, leaderboard sorted set. In-memory fallback for local/offline.
• Scoring engine (lib/server/scoring.ts) — structural + fuzzy Levenshtein field matching for OCR tolerance. Answer keys are server-only.
• Defenses — lib/ratelimit.ts (per-participant breaker), lib/geo.ts (geo-gating), lib/tokens.ts (HMAC tokens + per-team scrape nonce).