Attica traffic lab

A research lab for understanding and predicting road traffic in Attica (Athens and Piraeus, Greece) from open and free-tier data: a fleet of small collectors that record feeds which keep no history, and analysis code that turns them into road speeds, timetable delays, signal timings, simulations, forecasts and structured traffic rules.

Athens publishes no live road-speed feed, no signal timing plans and no traffic rules as data; its open loop-detector feed stopped in 2026. The lab tests how far public bus GPS, municipal counters, drone datasets, official decisions and a local LLM can close that gap.

Status: research code, first pass. The numbers below come from a few days of collection (September 2026) and are labelled with their sample sizes. Nothing here is production software.

Contents

Methods and results

1. Traffic-light cycles from bus GPS, validated on drone data

analysis/signals.py, analysis/pneuma_signals.py, analysis/pneuma_busval.py, analysis/pneuma.py

OASA buses report their position every ~20–30 s. For every OSM-signalised junction on a bus route, a pass that has two standing fixes 5–120 m before the junction and then a fix past it is a "queued at red, then released" event. Events are pooled per junction × approach across all lines, corrected for the release lag of the queue position, and their times are folded modulo candidate cycles C = 50…160 s. The C that maximises the weighted mean resultant length R (Rayleigh statistic) is the cycle; a near-tie rule prefers common plan lengths {80, 90, 100, 120, 135} s.

Ground truth comes from the pNEUMA drone dataset (EPFL, 2018, ~1.0 M vehicle crossings):

• 82 junctions in central Athens have a clear fixed-time cycle, in two coherent zones: 53 × 90 s (Patision / Panepistimiou / Omonia axis) and 28 × ≈135 s (the Exarchia-side grid). Median green per approach 51 s, stable to ±2 s across flights (median IQR 2.1 s). Absolute offsets are not recoverable: each 30-minute drone slot has its own clock.
• Degrading pNEUMA buses to OASA-like fixes (every 15/20/30 s, 3–5 m noise) calibrated the detector: the release lag is +5 to +9 s and grows with queue position; a standing threshold of 8 m yields ~45 % more events than 5 m at equal accuracy; pooling must be per approach, not per line.
• On two real days of bus data the recalibrated detector found 133 approach series (90 s × 84, 80 s × 10, 120 s × 8). At the 6 approaches that overlap the drone footprint, 4 agree with the drone truth, including all 3 with ≥ 15 events; both misses are low-n approaches in the 135 s zone fed by the same lines.
• Statistics caveat. The p-value printed by signals.py is a Greenwood–Durand series truncated after its first correction term; it goes negative for strong peaks and ignores that the best of 111 candidate cycles was picked. Use Zar's (1999) approximation p = exp(sqrt(1 + 4n + 4(n² − R_n²)) − (1 + 2n)) with R_n = nR, and a look-elsewhere (max-over-grid) null. With a Monte-Carlo max-R null, 96 of 133 series stay significant at 0.05, and every exotic cycle (59, 94, 109, 131 s …) falls out, including both overlap misses: the detector knows when it does not know. (These two corrections were computed in a separate check and are not yet patched into signals.py.)
• The fold is a clock-precision instrument: peaks in R(C) are ~0.05 s wide over two days, so Athens controllers apparently run exact integer cycles phase-locked to wall-clock time.

2. Buses as road-speed sensors, and timetable delays

analysis/osm_edges.py, analysis/bus_speeds.py, analysis/calibrate.py, analysis/profiles.py, analysis/gtfs_delay.py, analysis/pick_calib_points.py

• Map-matching. 373 OASA route shapes are densified every 10 m and snapped to the 924,662-edge OSM drivable network (KD-tree, ≤ 25 m, bearing ≤ 50°, oneway-aware, majority smoothing). Each fix is linear-referenced on its own route with continuity to the previous fix (so loops resolve); consecutive fixes 20–240 s apart give a speed that is spread over the covered edges → edge × direction × 15-min speeds.
• On 24/09/2026: 740,587 bus fixes, 2.08 M speed cells on 42,857 edges (1,248 km); 99 % of fixes projected; cross-route agreement on shared cells r = 0.73.
• Bus → car calibration. ln(car) = a + b·ln(bus) against TomTom Flow Segment readings at 24 well-covered edges (468 pairs, 21 points, 3 days): leave-one-day-out MAE 14.5 → 10.1 km/h (pooled model ln car = 1.413 + 0.679 ln bus). Per-class fits are not yet stable. No TomTom readings are published here.
• Delays. Observed trips are matched 1:1 to the day's schedule (OASA daily schedule + GTFS stop profiles, Hungarian assignment; 86 % of fixes matched): 287,117 stop-level delay rows, median delay +87 s (p10 −313 s, p90 +901 s). Buses run faster than the timetable between stops (actual/scheduled running time 0.92) yet arrive late, so lateness is mostly carried from terminal departures. Hours affected by collection gaps must be masked: a poller gap looks exactly like a service collapse downstream.

3. What-if simulation with SUMO

analysis/sim_piraeus.py, analysis/trafficsim/, analysis/demand_v2.py, analysis/build_ff_net.py, analysis/junction_delay.py, analysis/sim_eval.py, analysis/sumo_smoke.sh, analysis/simtools/

SUMO 1.27 mesoscopic models built from OSM (netconvert with guessed/joined signals), demand from an in-house hourly activity model (population, jobs, POIs, visitors, freight) with one global scale k, calibrated against aggregated TomTom congestion levels.

• First pass (port-corridor sub-network west of central Piraeus, 15.2k passenger edges, 186 signals): network speed-ratio MAE 0.130 at k = 0.9 for the 06:00–10:00 peak of 24/09/2026. Single-seed scenarios: +400 port trucks/h +6.7 % vehicle-hours; one lane of a port-access arterial closed for an hour +13.5 %. A street-market closure scenario silently did nothing in meso (fixed later with a patched network).
• The honest lesson from the wider 143k-edge Athens–Piraeus net: speed-only calibration keeps preferring the lowest demand (an under-loaded network always "matches" free-flow-ish speeds), 382k teleports swamp the run, and scenario effects shrink to within ±1.5 % (trucks +1.4 %, lane closure −0.1 %) once vehicle-hours are measured at a common horizon instead of over completed trips. Speeds alone cannot identify demand; counts are needed.
• Follow-ups in the code: a repaired free-flow network (short edges into joined junctions had been crushed to ~11 km/h, the root cause of the gridlock), DUA routing, a census-based demand model (v2), HCM-6 junction delays, a health gate that rejects runs with > 1 % jam teleports, config-driven multi-seed scenario runs with paired confidence intervals, and Optuna calibration of meso junction penalties.

4. Forecasting port-gate traffic (LightGBM)

analysis/piraeus_model.py, analysis/f1_forecast.py; figures in results/piraeus_forecast/

Hourly vehicle counts from five Municipality of Piraeus counters (10-min bins since 01/2026), rolling-origin test over the last 8 weeks: LightGBM (Poisson; calendar, Greek holidays, school periods, events, same-hour-last-week lag) reaches WAPE 21.4 % vs 26.3 % for "same hour last week" and 35.6 % for the hour-of-week mean. The gap is largest in the 15-August week (26.8 % vs 40.8 %). Weather makes it slightly worse, and football matches at the nearby stadiums do not move these counters (median ratio 0.989 within ±3 h of 41 games): the gates follow port operations.

5. A local LLM reads Greek traffic decisions

collectors/diavgeia.py, collectors/rules_hunter.py, collectors/events/ (events.py, events_llm.py, events_geo.py), analysis/rules_edges.py, analysis/build_street_index.py, analysis/diavgeia_orgs.py, analysis/bench_gliner.py, analysis/qwen_scout.py, analysis/scout_verify.py

• Harvests 21,241 Attica traffic decisions (2015 → 2026) from the Diavgeia transparency portal plus police traffic-arrangement releases.
• A local Qwen3.8-27B with a strict JSON schema (temperature 0, response_format: json_schema) extracts dated disruption events (events_llm.py: schema + system prompt) and standing city rules (rules_hunter.py: 19 rule types, status, vehicles, days, time windows, validity, confidence and a ≤ 200-char verbatim quote).
• Every rule's quote is verified accent- and case-insensitively against the source text; a failed quote multiplies the model's confidence by 0.8. A stdlib geocoder with a light Greek stemmer resolves "street X between A and B" over 75k OSM street names.
• So far 882 rules from the first 145 of 466 corpus documents; 81 % of quotes verified verbatim; 22/22 known-fact checks passed; spot-check precision 7/7 at confidence ≥ 0.8 with a verified quote (7/10 overall — tiny samples).
• Police enforcement (checkpoints, alcohol tests, speed traps, camera enforcement) is filtered out twice, by regex before the LLM and inside the prompt. The project never maps enforcement.

6. Sun glare per road and direction

analysis/sunglare.py

NOAA solar-position algorithm (with refraction) against the bearing of every main-road edge direction in Attica (238,686 edges, 384,754 edge-directions): glare = sun 0–20° above the horizon and within ±25° of the heading. Glare hits 2.34 % of edge-direction × week × 15-min slots, and winter has ~24 % more glare slots than midsummer. First check against evening bus speeds: no measurable effect (ratio 0.997 vs 0.991) — an honest null.

Other pieces

Activity model (activity_*.py, census_pop.py, elstat_pop.py, visitors.py, poi_extract.py, overture_*.py, fsq_attica.py), static edge features (edge_features.py, eff_lanes_model.py, pneuma_lanes.py), TomTom tile/incident map-matching to OSM edges (tiles_to_edges.py, incidents_to_edges.py), street markets (laiki_edges.py), air-quality patterns as long-history ground truth (airq*.py: NOx increments track traffic per day, not per hour), micromobility (gbfs_report.py), Mapillary vs OSM signals (mapillary_signals.py), OSRM time-sliced matrices (osrm_speeds.py, osrm_matrix.py, zone_matrix.py).

Results in this repository

How to run

Requirements: Python ≥ 3.12. Collectors: pip install -r requirements-collectors.txt (mostly stdlib). Analysis: pip install -r requirements-analysis.txt (includes eclipse-sumo).

cp .env.example .env            # fill in only the keys you have; never commit .env
set -a; . ./.env; set +a

# collector host
python3 collectors/keycheck.py                 # checks each key with one cheap call, never prints keys
python3 collectors/collector.py                # bus GPS + motorway detectors + weather (runs forever)
deploy/run_collector.sh tomtom_tiles.py status # any other collector, with .env loaded
# see deploy/crontab.example and deploy/*.service for the full schedule

# analysis machine ($DATA_DIR holds everything heavy)
deploy/sync_from_collector.sh                  # nightly pull of finished collector days
cd analysis
python osm_extract.py $DATA_DIR/greece-latest.osm.pbf $DATA_DIR/osm
python osm_edges.py $DATA_DIR/greece-latest.osm.pbf $DATA_DIR/osm/edges.parquet
python bus_speeds.py all && python signals.py 10
python pneuma.py convert --hz 5                # pNEUMA zip -> parquet (needs the dataset, see DATA.md)
python pneuma_signals.py crossings && python pneuma_signals.py timing
python -m pytest -q                            # unit tests (< 2 s); -m smoke runs a small SUMO net

Configuration is by environment variable only: COLLECTOR_DIR, DATA_DIR, COLLECTOR_HOST / COLLECTOR_PATH, OASA_AOI / OASA_FOCUS, SIM_AREA / SIM_BBOX, SIM_VENV_BIN, LLM endpoints and the API keys listed in .env.example. Heavy pipelines need their input data (see DATA.md) and will not run on a fresh checkout.

Not included

• No data of any kind: no collected SQLite stores, no parquet warehouse, no OSM extracts, no pNEUMA files, no TomTom / HERE / airport data, no SUMO networks or runs, no model files. See DATA.md for where to get each source.
• No keys or credentials. Collectors read keys from the environment.
• No personal or location-specific configuration. The original deployment's areas of interest, focus routes, calibration points, probe/reference routes and machine names are replaced by neutral defaults (central Athens and central Piraeus) and environment variables.
• No enforcement mapping. Speed-camera extraction and red-light-camera lists are deliberately absent.
• Private home-server integrations (dashboard patches, notification hooks) and one airport source whose terms allow personal viewing only were removed.

Data sources and licences

See DATA.md for every source with its licence/terms and required citation. In short: OpenStreetMap (ODbL), pNEUMA (CC BY 4.0, cite Barmpounakis & Geroliminis 2020), OASA telematics and data.gov.gr open data, Diavgeia public acts, Open-Meteo/ERA5 (CC BY 4.0), EEA air quality; TomTom, HERE and airport data were used only under their free-tier terms and are not redistributed.

Authorship and credits

Research direction, questions, source selection, rules and verification: John Vrachnis, as an MSc-level research project at the University of Piraeus (Department of Informatics). The code was written with AI assistance: Claude Code agents implemented most scripts under John's direction and review, and a local Qwen model performs the LLM extraction steps. Git history of the original working copy does not record per-line authorship.

Data credits: "Data source: pNEUMA – open-traffic.epfl.ch" (Barmpounakis & Geroliminis, 2020); © OpenStreetMap contributors; OASA; Municipality of Piraeus / data.gov.gr; Diavgeia; Open-Meteo.com; EEA.

References

• Barmpounakis, E., & Geroliminis, N. (2020). On the new era of urban traffic monitoring with massive drone data: The pNEUMA large-scale field experiment. Transportation Research Part C, 111, 50–71.
• Zar, J. H. (1999). Biostatistical Analysis, 4th ed. (Rayleigh test p approximation.)
• Greenwood, J. A., & Durand, D. (1955). The distribution of length and components of the sum of n random unit vectors. Annals of Mathematical Statistics, 26(2), 233–246.
• Fayazi, S. A., Vahidi, A., Mahler, G., & Winckler, A. (2015). Traffic signal phase and timing estimation from low-frequency transit bus data. IEEE Trans. ITS, 16(1), 19–28.
• Lopez, P. A., et al. (2018). Microscopic Traffic Simulation using SUMO. IEEE ITSC.
• Ke, G., et al. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. NeurIPS.
• Kuhn, H. W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2.
• NOAA Global Monitoring Laboratory, Solar Calculation Details (after Meeus, Astronomical Algorithms).

License

Code: MIT (see LICENSE). Data: not included; each source keeps its own licence (DATA.md).