Langdev libraries for Rust

• Lexers
• Parsers
• Codegen
• String interning
• Just-in-time compilation
• Compiler framework
• Error reporting
• Language server protocol
• Testing
• Incremental compilation
• Floats/Ints/Bools
• Binary & object file parsing, generating and processing
• Solvers
• CLI
  • Configuration
• Repl
• String handling
• Syntax trees
• Pretty printing
• Variable binding
• Caching
• WASM
• Logging
• Storage
  • Disjoint-sets
• Incremental Analysis
  • Timely Dataflow Resources
• Fuzzing
  • Fuzzing Resources
• Graphs
• Type Checking
• Peephole Optimization
• Garbage Collection
• Guides & Resources
  • Type Checking
  • Inlining
  • RVSDG
  • Equality Saturation
• Uncategorized

Lexers

• logos Logos has two goals: To make it easy to create a Lexer, so you can focus on more complex problems and to make the generated Lexer faster than anything you'd write by hand
• lrlex A replacement for lex/flex that generates Rust code
• lexgen A fully-featured lexer generator, implemented as a proc macro.

Parsers

• lalrpop A convenient LR(1) parser generator
• nom A byte-oriented, zero-copy, parser combinators library
• combine Fast parser combinators on arbitrary streams with zero-copy support.
• pom PEG parser combinators using operator overloading without macros.
• peg A simple Parsing Expression Grammar (PEG) parser generator.
• glue Glue is a parser combinator framework for parsing text based formats, it is easy to use and relatively fast too
• pratt A general purpose pratt parser for Rust
• pest A general purpose parser written in Rust with a focus on accessibility, correctness, and performance using parsing expression grammars (PEG)
• lrpar A Yacc-compatible parser.
  • nimbleparse_lsp An LSP server for quickly developing lrpar parsers, which parses input buffers on grammar change bypassing code generation.
• tree-sitter A parser generator for building fast editor-friendly parsers with features like streaming, incremental, resumable (re)parsing, with timeouts and cancellations, and advanced syntax node queries.
• rowan Generic lossless syntax trees
  • ungrammar A DSL for specifying concrete syntax trees, see this introductory post
• cstree A fork of rowan with threadsafe syntax trees and built-in source string interning
• chomp Chomp is a fast monadic-style parser combinator library designed to work on stable Rust
• oak A typed parser generator embedded in Rust code for Parsing Expression Grammars
• chumsky A friendly parser combinator crate that makes writing LL(k) parsers with error recovery easy
• rust-sitter A package for defining tree-sitter grammars alongside Rust logic

Regular expressions

• regex An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs.
• regress A regular expression engine targeting EcmaScript syntax.
• onig Rust-Onig is a set of Rust bindings for the Oniguruma regular expression library.

Codegen

• cranelift Cranelift is a low-level retargetable code generator
• llvm-sys Bindings to LLVM's C API
• iced-x86 A blazing fast and correct x86/x64 disassembler, assembler and instruction decoder
• llama Friendly LLVM bindings
• inkwell Inkwell aims to help you pen your own programming languages by safely wrapping llvm-sys
• llvm-ir LLVM IR in natural Rust data structures
• llvmenv Manage multiple LLVM/Clang builds
• walrus A library for performing WebAssembly transformations
• cilk Toy Compiler Infrastructure influenced by LLVM written in Rust.

String interning

• lasso A multithreaded and single threaded string interner with a minimal memory footprint and arena allocation
• string-interner A data structure to cache strings efficiently
• simple-interner A simple append-only interner
• string_cache A string interning library for Rust, developed as part of the Servo project

Just-in-time Compilation

• iced-x86 A blazing fast and correct x86/x64 disassembler, assembler and instruction decoder
• masm-rs A JSC/SpiderMonkey like macro assembler
• b3 Rust port of B3, LLVM-like backend
• jit (LibJIT) Just-In-Time Compilation in Rust using LibJIT bindings
• lightning-sys GNU lightning bindings for rust
• gccjit Higher-level Rust bindings for libgccjit
• cranelift Cranelift is a low-level retargetable code generator
  • cranelift-jit-demo A JIT compiler and runtime for a toy language, using Cranelift
• dynasm A Dynamic assembler written in Rust for Rust

Compiler Framework

• xrcf The eXtensible and Reusable Compiler Framework (xrcf) is a library for building compilers

Error reporting

• codespan-reporting Beautiful diagnostic reporting for text-based programming languages
• codespan Data structures for tracking locations in source code
• text-size A library that provides newtype wrappers for u32 and (u32, u32) for use as text offsets
• ariadne A fancy diagnostics & reporting crate
• miette Fancy diagnostic reporting library and protocol for us mere mortals who aren't compiler hackers

Language server protocol

• lsp-types Types for interaction with a language server, using VSCode's Language Server Protocol
• tower-lsp Language Server Protocol implementation based on Tower
• codespan-lsp Conversions between codespan types and Language Server Protocol types
• lsp-server A generic LSP server scaffold

Testing

• goldentests A golden file testing library where tests can be configured within the same test file
• lang_tester Concise language testing framework for compilers and VMs (Linux only)
• libtest-mimic Dynamically construct a test-harness that looks and behaves like Rust's built-in test harness
• compiletest_rs The compiletest utility from the Rust compiler as a standalone testing harness
• insta A snapshot testing library for Rust
• k9 Snapshot testing and better assertions
• bulk_examples_generator A tool created in Rust for create dozens/hundreds/thousands/millions of random examples based on a pest grammar (PEG)

Unicode

• unicode-xid Determine if a char is a valid identifier for a parser and/or lexer according to Unicode Standard Annex #31 rules.
• unicode-normalisation Unicode character composition and decomposition utilities as described in Unicode Standard Annex #15
• unicode-segmentation Iterators which split strings on Grapheme Cluster or Word boundaries, according to the Unicode Standard Annex #29 rules
• unicode-width Determine displayed width of char and str types according to Unicode Standard Annex #11 rules
• lexical-sort Lexicographical sorting of unicode strings

Incremental compilation

• salsa A generic framework for on-demand, incrementalized computation
• verde A refreshingly simple incremental computation library

Floats/Ints/Bools

• lexical Lexical, to- and from-string conversion routines (Fast lexical conversion routines for both std and no_std environments)
• lexical-core Lexical, to- and from-string conversion routines (Low-level, lexical conversion routines for use in a no_std context)
• lexical_bool A bool-like type that can be parsed from a string
• ryu A Rust implementation of the PLDI'18 paper Ryū: fast float-to-string conversion by Ulf Adams
• ryu-js Ryū-js is a fork of the ryu crate adjusted to comply to the ECMAScript number-to-string algorithm
• hexponent C11 compliant hex float parsing
• ordered-float Total ordering on floats
• half A half-precision floating point f16 type for Rust implementing the IEEE 754-2008 standard
• f128 Bindings to the gcc quadmath library
• approx Approximate floating point equality comparisons and assertions

Binary & object file parsing, generating and processing

• goblin An impish, cross-platform, ELF, Mach-o, and PE binary parsing and loading crate
• gimli A library for reading and writing the DWARF debugging format.
• faerie ELF and Mach-o native binary object file emitter
• object A unified interface for reading and writing object file formats.
• elf A pure-rust library for parsing ELF files
• elfkit An elf parser and manipulation library in pure rust

Symbolic Execution

• haybale Symbolic execution of LLVM IR

Solvers

• rsmt2 Wrapper for SMT-LIB 2 compliant SMT solvers.
• z3 A high-level rust bindings for the Z3 SMT solver from Microsoft Research
• z3-sys Low-level bindings for the Z3 SMT solver from Microsoft Research
• z3_ref A high level interface to the Z3 SMT solver
• z3d Z3 DSL interface for Rust
• boolector Safe high-level bindings for the Boolector SMT solver
• boolector-sys Low-level bindings for the Boolector SMT solver
• smt2utils Libraries and tools for the SMT-LIB-2 standard
  • smt2parser A generic parser for SMT2 commands, as specified by the SMT-LIB-2 standard
  • smt2proxy An experimental binary tool to intercept and pre-process SMT2 commands before they are send to an SMT solver
  • z3tracer An experimental parser for Z3 tracing logs obtained by passing trace=true proof=true to Z3
• good_lp Mixed Integer Linear Programming for Rust, with an user-friendly API. This crate allows modeling LP problems, and lets you solve them with various solvers.
• Axiom Profiler A tool for visualising, analysing and understanding quantifier instantiations made via E-matching in a run of an SMT solver
• [2017][PDF] Counterexample Guided Inductive Optimization based on Satisfiability Modulo Theories

CLI

• structopt Parse command line arguments by defining a struct
• clap A simple to use, efficient, and full-featured Command Line Argument Parser
• pico-args An ultra simple CLI arguments parser
• argh A derive-based argument parser optimized for code size

Configuration

• etcetera A library that aims to allow you to determine the locations of configuration, cache and data files for your application. Existing Rust libraries generally do not give you a choice in terms of which standards/conventions (Etcetera calls these ‘strategies’) they follow. Etcetera, on the other hand, gives you the choice
• toml A TOML decoder and encoder for Rust (optionally uses serde)
• envy Deserialize environment variables into typesafe structs (uses serde)

Repl

• rustyline Rustyline, a readline implementation based on Antirez's Linenoise
• repl-rs Library to generate a REPL for your application
• termwiz Terminal Wizardry for Unix and Windows

String handling

• beef Faster, more compact implementation of Cow.
• smol_str Small-string optimized string type with O(1) clone
• smallstring 'Small string' optimization: store small strings on the stack using smallvec
• heck Heck is a case conversion library that exists to provide case conversion between common cases like CamelCase and snake_case. It is intended to be unicode aware, internally consistent, and reasonably well performing
• ropey Ropey is a utf8 text rope for Rust, designed to be the backing text-buffer for applications such as text editors. Ropey is fast, robust, and can handle huge texts and memory-incoherent edits with ease.

Syntax trees

• rowan Generic lossless syntax trees
• cstree A fork of rowan with threadsafe syntax trees and built-in source string interning

Pretty printing

• pretty Wadler-style pretty-printing combinators in Rust

Variable binding

• moniker An automagical variable binding library for tracking variables in scopes

Caching

• cached Caching structures and simplified function memoization

WASM

• wain WebAssembly interpreter written in Safe Rust with zero dependencies
• wasmer The high-level public API of the Wasmer WebAssembly runtime
• wasmtime High-level API to expose the Wasmtime runtime
• wasmtime-jit JIT-style execution for WebAsssembly code in Cranelift
• wasmlite-parser This crate parses WebAssembly modules and can generate LLVM IR based the data extracted from a module
• parity-wasm-cp WebAssembly binary format serialization/deserialization/interpreter
• substrate-wasm-builder A utility for building WASM binaries
• walrus A library for performing WebAssembly transformations

Logging

• tracing Application-level tracing for Rust
  • tracing-tree A Tracing Layer which prints a tree of spans and events.
  • tracing-forest Preserving contextual coherence among trace data from concurrent tasks
  • tracing-timing Inter-event timing metrics on top of tracing
  • tracing-coz Rust-tracing support for the coz Causal Profiler
  • tracing-flame A tracing Layer for generating a folded stack trace for generating flamegraphs and flamecharts with inferno
  • test-env-log A crate that takes care of automatically initializing logging and/or tracing for Rust tests.
  • tracing-unwrap This crate provides .unwrap_or_log() and .expect_or_log() methods on Result and Option types that log failed unwraps to a tracing::Subscriber
• log A Rust library providing a lightweight logging facade
• slog An ecosystem of reusable components for structured, extensible, composable and contextual logging for Rust

Storage

• slotmap Containers with persistent unique keys to access stored values: Insertion, deletion and access all take O(1) time with low overhead.
• indexmap A hash table with consistent order and fast iteration
• vecmap Vec-based Map and Set data structures

Disjoint-sets

• union-find Struct and methods for union-find operation
• ena An implementation of union-find in Rust; extracted from (and used by) rustc

Incremental Analysis

• timely A low-latency cyclic dataflow computational model, introduced in the paper Naiad: a timely dataflow system
  • differential-dataflow A data-parallel programming framework designed to efficiently process large volumes of data and to quickly respond to arbitrary changes in input collections, an implementation of differential dataflow over timely dataflow on Rust.
  • dogsdogsdogs Worst-case optimal joins and delta queries in differential dataflow
  • dataflow-join A streaming implementation of Ngo et al's GenericJoin in timely dataflow
  • timely-sort An cache-aware implementation of radix sort in Rust
  • diagnostics Diagnostic tools for timely-datafflow and differential-dataflow computations
• differential-datalog A programming language for incremental computation based on timely-dataflow and differential-dataflow, similar to IncA or Souffle

Timely Dataflow Resources

• Naiad: a timely dataflow system
• Timely Dataflow Book
• Differential Dataflow
• Differential Dataflow Book
• Sudoku in Differential Dataflow: blog, video series
• An introduction to Timely Dataflow: part 1, part 2, part 3
• An introduction to Differential Dataflow: part 1, part 2
• Differential Dataflow meets Calculus

Fuzzing

• honggfuzz-rs Honggfuzz is a security oriented fuzzer with powerful analysis options that supports evolutionary, feedback-driven fuzzing based on code coverage
• cargo-fuzz A cargo subcommand for using libFuzzer
• libfuzzer-sys Barebones wrapper around LLVM's libFuzzer runtime library
• wasm-smith A WebAssembly test case generator
• fuzzcheck A structure-aware coverage-guided fuzzer
• libafl
• test-fuzz

Fuzzing Resources

• Fuzzing a parser
• Rust Fuzz Book
• Fuzzcheck guide
• The Fuzzing Book
• LibAFL guide

Graphs

• egg Egg is a flexible, high-performance e-graph library
  • Herbie's usage of egg to improve the accuracy of floating point calculations (language, rules, analysis)
  • Szalinski's usage of egg to simplify CAD programs (language, rules, appliers)
  • Glenside's usage of egg for optimizing tensor programs (language, rules, analysis)
• petgraph A graph data structure library. Provides graph types and graph algorithms
• timely A low-latency cyclic dataflow computational model, introduced in the paper Naiad: a timely dataflow system
• differential-dataflow A data-parallel programming framework designed to efficiently process large volumes of data and to quickly respond to arbitrary changes in input collections, an implementation of differential dataflow over timely dataflow on Rust.

Type Checking

• rusttyc An interface that allows for an intuitive translation of inference rules based on a Hindney-Milner-like bounded type meet-semilattice into rust code
• polytype A Hindley-Milner polymorphic typing system

Peephole Optimization

• egg Egg is a flexible, high-performance e-graph library
• peepmatic A DSL for peephole optimizations and compiler for generating peephole optimizers from them

Garbage Collection

• broom An ergonomic tracing garbage collector that supports mark 'n sweep garbage collection
• gc A simple tracing (mark and sweep) garbage collector for Rust
• shredder Garbage collection as a library for Rust
• comet A garbage collection library for implementing VMs in Rust
• mmtk Memory Management ToolKit (MMTk) for portable & efficient memory management in language VMs

Guides & Resources

• Making a language in Rust A series about making a programming language called Eldiro using the Rust programming language
• Crafting Interpreters Implementing everything from parsing to garbage collection to bytecode VMs, highly reccomended as an introduction to language development (Homepage)
• Create Your Own Programming Language with Rust
• Where to Start Hand-Writing a Parser (in Rust)
• Writing a simple parser in Rust
• GC and Rust part 0, part 1 and part 2
• Matklad on pratt parsing: Simple but Powerful Pratt Parsing and From Pratt to Dikjstra
• Pure AST based linting sucks
• LLVM's Kaleidoscope in Rust The Kaleidoscope Language tutorial, showing how to implement a simple language using LLVM in Rust
• Where to Start Hand-Writing a Parser (in Rust) An introduction to programming language parsing in which we hand-write a parser and run it on some real input
• x86 and amd64 instruction reference
• X86 Opcode and Instruction Reference
• Linear Scan Register Allocation on SSA Form
• A bibliography of papers related to symbolic execution
• Instruction Latencies, Throughput and Micro-Operation Breakdowns for Intel, AMD and VIA CPUs
• [1995][PDF] Symbolic Range Propagation
• [2001][PDF] Efficient Symbolic Analysis for Optimizing Compilers
• [2021][Video] A New Approach to Removing Redundant Conditions in LLVM
• [2017][Video] Polyhedral Value & Memory Analysis
• Simplify: A Theorem Prover for Program Checking
• Elegant and performant recursion in Rust
• Speculation in JavaScriptCore
• Whippet: A new GC for Guile

Type Checking

• Type Checking in less than 100 lines of Rust (Credit to @zesterer)
• Bidirectional Type Checking in ~200 lines of OCaml (Written in OCaml, credit to @mb64)
• [2020][PDF] Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism
• Implementation of "Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism" (Written in Rust, from @segeljakt)
• Implementation of "Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism" (Written in Rust, from @JDemler)
• A bidirectional type inference system loosely based on Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism. (Written in Haskell)
• Didactic implementation of the type checker described in "Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism" (Written in OCaml)
• An implementation of a predicative polymorphic language with bidirectional type inference and algebraic data types (Written in Haskell)

Inlining

• Inlining Decisions in Visual Studio
• C++ Inliner Improvements: The Zipliner
• John Carmack on Inlined Code
• How RyuJIT inlines a function (heuristics)
• Smarter C/C++ inlining with attribute((flatten))
• Do compilers take inline as a hint?
• cmd/compile: improve inlining cost model
• What does 'inline' mean?
• Zebu VM check_should_inline_func()
• Warp: Improved JS performance in Firefox 83
• When Short Methods Pay Off: JIT Inlining
• cross-module inlining in guile -- wingolog
• Inline In Rust
• It’s Not Always iCache
• [2003][PDF] Adaptive Online Context-Sensitive Inlining
• [2003][PDF] To Inline or Not to Inline. Enhanced Inlining Decisions
• [2004][PDF] Fast and Effective Procedure Inlining
• [2016][PDF] CSE P 501 - Compilers. Inlining and Devirtualization
• [2017][PDF] TurboFan Inlining Heuristics
• [2019][PDF] Guiding Inlining Decisions Using Post-Inlining Transformations
• [2019][PDF] An Optimization-Driven Incremental Inline Substitution Algorithm for Just-in-Time Compilers
• [2021][PDF] Inlining for Code Size Reduction

RVSDG

• [2020][PDF] RVSDG: An Intermediate Representation for Optimizing Compilers
• [2018][PDF] RVSDG: An Intermediate Representation for the Multi-Core Era
• [2015][PDF] Perfect Reconstructability of Control Flow from Demand Dependence Graphs
• Principles, Techniques, and Tools for Explicit and Automatic Parallelization
• [Slides] Compiling with the Regionalized Value State Dependence Graph
• JLM RVSDG Implementation (C++)
• JIVE RVSDG Graph Implementation (C++)
• RVSDG Viewer (C++)
• Cranial Coitus RVSDG Brainfuck Optimizer (Rust)
• [2018][PDF] Semantic Reasoning About the Sea of Nodes
• [1994][PDF] Value dependence graphs: representation without taxation
• [2016][PDF] Value State Flow Graph: A Dataflow Compiler IR for Accelerating Control-Intensive Code in Spatial Hardware
• Numba Python JIT RVSDG Optimizer
• [Archived] Rain IR A dependently-typed RVSDG with liftetimes, implemented in Rust

Equality Saturation

• [2010][PDF] Equality Saturation: A New Approach to Optimization
• [2020][PDF] egg: Fast and Extensible Equality Saturation
• [2021][PDF] Rewrite Rule Inference Using Equality Saturation
• [2021][PDF] Sketch-Guided Equality Saturation: Scaling Equality Saturation to Complex Optimizations in Languages with Bindings
• [2021][PDF] Caviar: An E-graph Based TRS for Automatic Code Optimization
• [2002][PDF] SPORES: Sum-Product Optimization via Relational Equality Saturation for Large Scale Linear Algebra
• [2021][PDF] Relational E-Matching
• [2021][PDF] Rewrite Rule Inference Using Equality Saturation

Uncategorized

• Turns are Better than Radians An interesting blog post about using turns instead of pi or tau for trigometric calculations, could be inspiration for some interesting code transformations
• Algorithms for Modern Hardware (russian version) Low-level performance tuning for modern hardware