semantic-editor (semedit)

Intent-driven code editing for AI agents: LLMs plan intent, host compilers execute zero-token AST refactorings (up to 100x fewer tokens, 10x faster).

semedit separates semantic intent (decided by the LLM) from mechanical syntax transformation (executed by local host CPU compilers, LSPs, and AST tools).

---

1. Problem Statement

Modern coding agents rely on probabilistic token prediction to perform deterministic text manipulation:

• Token Asymmetry & Churn: Renaming a symbol across 10 files or extracting a helper function forces the LLM to re-emit hundreds of lines of diffs.
• Context Exhaustion: Large diffs consume context window budget, accelerating context compaction and degrading the model's reasoning capacity for subsequent turns.
• Fragile Transitions: Line-based search/replace tools frequently fail due to whitespace shifts, unclosed braces, or partial generation, triggering expensive remote repair loops.
• Provider & Infrastructure Drift: Hosted model performance fluctuates with diurnal traffic cycles, silent provider-side quantization adjustments, KV-cache eviction policies, and API congestion. Offloading mechanical edits to the host CPU removes this variance from routine syntax operations.

---

2. Core Thesis & The Inverted Model

LLMs should issue semantic editing commands, not generate textual patches.

Traditional Agent Flow (Fragile Text Loop):
LLM Plan ──► LLM Streams 1000s of Diff Tokens ──► Network Latency ──► Disk Edit ──► [Syntax Failure / Retry]

Semantic Agent Flow (Deterministic Execution):
LLM Plan ──► LLM Emits 1 Intent (~20 tokens) ──► Host Tooling Transforms Code ──► Compiler Reports Diagnostics

Established local development toolchains (compiler typecheckers, language servers such as gopls, rust-analyzer, tsserver, jdtls, and AST utilities) already perform precise structural modifications deterministically on the host CPU.

semedit bridges AI agents directly to these engines.

---

3. Core Hypotheses

1. Efficiency: Local CPU code manipulation executes in milliseconds and consumes near-zero remote tokens, costing fractions of a cent per operation.
2. Determinism: Semantic transformations preserve syntactic validity across state transitions ($S_n \to S_{n+1}$). Syntax errors and diff-application drift are eliminated for supported operations.
3. Provider & Network Resiliency: Offloading editing to the host CPU protects agent sessions from remote API throttling, timeouts, dropped connections, and provider-side inference degradations.
4. Economics & Planning Horizon Scaling:
   • Preserves Context Budget: Text diffs consume context proportional to file and change size ($\mathcal{O}(\text{diff size})$). Semantic commands scale with intent ($\mathcal{O}(1)$ tokens per edit), preserving context for architecture and reasoning.
   • Lowers Viability Threshold: Makes micro-refactoring (renaming, inlining, stubbing) economically viable to delegate.
   • Preserves Flow State: Millisecond-range local execution maintains real-time pair programming cadence.
   • Bounded Blast Radius: Tooling alters only targeted symbols, eliminating accidental deletions of unrelated lines or comments.

Efficiency & Turn Mechanics (The "Up to 100x / 10x" Rationale)

The claim of up to $100\times$ token reduction and $10\times$ lower latency is derived analytically from the token mechanics of agentic tool loops and observed multi-file refactoring traces:

[!NOTE] Empirical Validation: These metrics are analytical upper bounds derived from standard multi-file refactoring operations. Empirical verification across heterogeneous harnesses (Antigravity, Claude Code, direct API), models, and reasoning levels is tracked under RQ-0014 and RQ-0015.

---

4. Value-Add on Top of Existing Tooling

Why not just have the LLM call gopls, rust-analyzer, or ast-grep directly?

Existing language servers and CLI tools were built for interactive human IDE sessions or static CI rules, not autonomous AI agents. semedit provides the missing coordination layer:

---

5. Architecture

semedit is not a new parser, AST library, or rewrite engine. It is strictly an orchestration broker and adapter layer that coordinates existing language servers and utilities.

┌────────────────────────────────────────────────────────────────────────┐
│                          AI Agent / LLM                                │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ High-level Intent (CLI or MCP)
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│                   Semantic Broker (`semeditd`)                         │
│                                                                        │
│  ┌───────────────────────┐             ┌────────────────────────────┐  │
│  │ Symbol Resolver       │             │ Execution Pipeline         │  │
│  │ (Tree-sitter queries) │             │ (Format, Verify, Snapshot) │  │
│  └───────────┬───────────┘             └─────────────┬──────────────┘  │
└──────────────┼───────────────────────────────────────┼─────────────────┘
               ▼                                       ▼
┌────────────────────────────────────────────────────────────────────────┐
│                          Backend Adapters                              │
├──────────────────────────┬─────────────────────────┬───────────────────┤
│ Tier 1: Compiler / LSP   │ Tier 2: Semantic LST    │ Tier 3: Error-    │
│ (Type-Checked Graph)     │ (Language Frameworks)   │ Tolerant CST      │
├──────────────────────────┼─────────────────────────┼───────────────────┤
│ • Go: gopls              │ • Java: OpenRewrite     │ • ast-grep        │
│ • Rust: rust-analyzer    │ • Python: Rope / LibCST │ • Tree-sitter     │
│ • TS: tsserver           │ • Elixir: Sourceror     │ • Comby           │
│ • Java: Eclipse jdtls    │                         │                   │
│ • Haskell: HLS           │                         │                   │
│ • Elm: elm-lang-server   │                         │                   │
└──────────────────────────┴─────────────────────────┴───────────────────┘

Four Primary Components

1. Agent Interface & Session Broker: Exposes high-level commands over both CLI (semedit rename) and MCP (semedit mcp). Manages daemon lifecycle and session state.
2. Symbol Resolver (Intent Addressing): Uses Tree-sitter queries to index symbols within files. Translates high-level agent targets (TokenService.Validate) into exact file/byte offsets required by LSPs.
3. Execution & Diagnostic Pipeline:
   • Applies the transformation to the file system.
   • Automatically executes language formatters (gofmt, rustfmt, prettier, ruff).
   • Runs compiler/typechecker passes and returns structured diagnostics directly to the LLM.
   • Provides on-demand snapshotting/undo if the agent or user decides to revert a change. (Edits do not automatically revert on compile errors, allowing multi-step refactorings that temporarily break code).
4. Backend Adapters:
   • Tier 1 (Compiler/LSP): Direct adapters for language servers (gopls, rust-analyzer, tsserver, jdtls, HLS, elm-language-server).
   • Tier 2 (Semantic Frameworks): Adapters for rich language-specific AST engines (OpenRewrite, Rope, Sourceror).
   • Tier 3 (Error-Tolerant CST): Fallback rewriters (ast-grep, Comby) used when the codebase contains syntax errors that prevent compiler tools from running.

---

6. Language Strategy & Ecosystem Roadmap

semedit does not implement a universal AST. Syntax trees cannot resolve cross-file types or inheritance. We build a universal intent broker that delegates to native compiler tools.

Tier 1: Initial Targets

Tier 2: Secondary & Functional Language Expansion

• Python: Rope + LibCST (heuristic type resolution; dynamic runtime bounds).
• C#: Roslyn (compiler-as-a-library; premier refactoring engine).
• Haskell: Haskell Language Server (HLS) for explicitly standalone, trusted, read-only hierarchical .hs symbol lookup. Project cradles, compilation, diagnostics, rename, retrie, and hlint remain unavailable.
• Elixir: ElixirLS / Lexical (LSP) for symbol navigation and refactorings paired with Sourceror / Igniter for lossless AST rewriting that preserves comments and formatting.
• Elm: elm-language-server paired with elm-review --fix and elm-format. Elm's compiler provides exceptionally deterministic error payloads, making the automated verification loop nearly zero-friction.
• Dart: Dart Analysis Server (explicit edit.getRefactoring RPC protocol).
• Scala: Metals + Scalafix.

Explicit Non-Goals (Phase 1)

• Concurrent Polyglot Language Servers: Running multiple language servers for different languages concurrently in a single workspace is postponed. Focus is strictly on single-language workspaces (though single-language multi-project repos are supported).
• C / C++: Complex compilation databases (compile_commands.json) and preprocessor macros require heavy per-project setup.
• Untyped languages without mature AST refactoring libraries: Low ROI for strict semantic validation.

---

7. How We Handle Go: Delegation vs. Reinvention

gopls already provides a CLI (gopls rename, gopls codeaction) and an experimental MCP server (gopls mcp).

We do not reinvent Go semantic refactoring. semedit delegates Go operations directly to gopls:

LLM Intent: `semedit rename --symbol "Server.Start" --to "Serve"`
                               │
                ┌──────────────┴──────────────┐
                │ If gopls MCP is active:     │
                │   Short-circuit to gopls    │
                │ Else:                       │
                │   Drive gopls CLI / LSP     │
                └──────────────┬──────────────┘
                               │
               Resolve symbol coordinates via AST
                               │
            Execute `gopls rename -w file.go:#offset`
                               │
            Auto-run `goimports` + report diagnostics

What semedit Adds on Top of gopls

1. Coordinate Resolution: gopls requires byte offsets or line/col numbers. semedit resolves symbol names to offsets via local Tree-sitter indexing.
2. Unified Agent Contract: The agent uses the same commands across Go, Rust, and TypeScript.
3. Execution Pipeline: gopls modifies files directly without running formatting, test validation, or reporting structured diagnostic summaries back to the agent.

---

8. Deliverables: Engine + Skills

A tool catalog alone is insufficient. Due to pre-training habits, frontier models default to generating raw diffs even when semantic tools are available.

semedit ships two synchronized artifacts:

1. The Execution Binary (semedit):
   • CLI Mode: semedit rename, semedit extract, semedit verify. Sub-millisecond shell commands for scripts and agents.
   • MCP Server Mode: semedit mcp. Exposes high-level tools to Claude Code, Cursor, Windsurf, Antigravity, and other MCP clients.
2. The Steering Layer (SKILL.md):
   • Agent rules and decision trees teaching models when to invoke semantic tools over text edits.
   • Prompts and heuristics for formulating symbol-based queries without hallucinating line numbers.

---

9. Proposed Project Layout

semedit/
├── cmd/
│   └── semedit/            # Main entry point (CLI and `semedit mcp` daemon)
├── internal/
│   ├── broker/             # Intent router and session management
│   ├── symbol/             # Tree-sitter locator (symbol query -> file:offset)
│   ├── pipeline/           # Staging, formatting, diagnostics, snapshotting
│   ├── fallback/           # Tier 3 syntax rewriters (ast-grep / comby integration)
│   └── adapters/           # Language-specific drivers
│       ├── golang/         # gopls CLI / LSP driver
│       ├── rust/           # rust-analyzer headless client
│       ├── typescript/     # tsserver JSON driver
│       ├── java/           # jdtls / OpenRewrite runner
│       ├── haskell/        # HLS / retrie runner
│       ├── elixir/         # ElixirLS / Sourceror runner
│       └── elm/            # elm-language-server runner
├── skills/
│   └── semedit/            # Agent skill definitions (SKILL.md, system rules)
├── Makefile                # Standardized check, test, build recipes
└── README.md

---

Appendix: Prior Art & Tooling Capability Matrix

The table below contrasts existing tools across the three tiers against semedit:

Detailed Notes on Tool Specializations

1. Why gopls / rust-analyzer are not competitors: semedit does not implement type-checkers or compiler front-ends. It uses gopls and rust-analyzer as backend execution workers. The value is bridging the gap between an LLM's natural way of thinking ("Rename function X") and the language server's rigid requirements ("Offset 1042 in buffer 3").
2. Why ast-grep is a partner, not a replacement: Tree-sitter and ast-grep are exceptionally good at parsing invalid code and finding structural patterns, but cannot distinguish between two methods with identical names belonging to different types across modules. semedit pairs ast-grep for local structural repairs with LSPs for project-wide semantic edits.
3. Why OpenRewrite complements interactive refactoring: OpenRewrite operates at the batch/recipe layer (e.g. migrating 500 files to a new logging framework). semedit delegates batch migration tasks in Java to OpenRewrite while using jdtls for granular, interactive single-step edits.
4. Functional & Pure Language Advantages (Haskell, Elixir, Elm):
   • Haskell: Enables equational rewriting via retrie: transformations can replace expressions according to algebraic laws while GHC guarantees semantic equivalence.
   • Elixir: Homoiconic syntax enables lossless AST manipulation via Sourceror and Igniter, allowing the agent to perform safe pattern replacements that respect comments and formatting.
   • Elm: The Elm compiler is famous for generating the most precise, human-readable, and machine-parsable error messages in software engineering. Verification and automated diagnostic repair loops are simpler in Elm than in virtually any other ecosystem.

For architectural invariants and historical decisions, see the Architecture Decision Records (ADRs). For unresolved spikes and open technical challenges, see the Research Questions Index. For development workflow, testing standards, and test authoring guides, see CONTRIBUTING.md.