Awesome Binary Parsing

A curated collection of tools and resources for parsing and analyzing binary data structures, such as file formats, network protocols or bitstreams.

Libraries and Tools by Programming Language

Python

• Construct: library for parsing and building of data structures (binary or textual). Define your data structures in a declarative manner
• Hachoir: view and edit a binary stream field by field. Long list of parsers for all kinds of formats
• Caterpillar: Python 3.12+ library to pack and unpack structurized binary data
• Scapy: send, sniff and dissect and forge network packets. Usable interactively or as a library
• Mr. Crowbar: Django-esque model framework for reading and writing binary file formats. Includes a suite of command-line tools for visualising and digging through binary data
• dissect.cstruct: structure parsing in Python made easy. Write C-like structure definitions and use them to parse binary data
• dpkt: fast, simple packet creation and parsing for the basic TCP/IP protocols. Pure Python, reads pcap/pcapng files

JavaScript

• Binary-parser: binary parser builder library which enables you to write efficient parsers in a simple & declarative way
• jBinary: High-level API for working with binary data.
• Binpat: Parse binary data using declarative patterns.
• restructure: declarative binary data encoding and decoding with support for C-like structures, pointers, arrays, bitfields and custom types. Used by fontkit and pdfkit

C/C++

• Hammer (C): bit-oriented parsing library
• FormatFuzzer (C++): framework for high-efficiency, high-quality generation and parsing of binary inputs
• Marpa (C/C++, Perl, Go): libmarpa (C)
• Wuffs: a memory-safe programming language (and a standard library written in that language) for Wrangling Untrusted File Formats Safely. Wrangling includes parsing, decoding and encoding.
• libtins (C++): crafting, sending, sniffing and interpreting raw network packets
• libcrafter (C++): high level library for C++ designed to create and decode network packets

Java

• Preon (Java): Bit syntax for Java. A declarative data binding framework for dealing with binary encoded data
• Apache Daffodil (Scala/Java, XML Schema): an open-source implementation of DFDL (Data Format Description Language) capable of describing many industry and military standards and parsing into a infoset, which is most commonly represented as either XML or JSON, and writing back to native format.

Go

• restruct: library for reading and writing binary data
• struc: better binary packing for Go; pack and unpack C-style structs from byte slices
• gopacket: packet processing library providing layered protocol decoding, pcap/pfring/afpacket bindings. Originally by Google

Rust

• Nom: Rust parser combinator framework
• Deku: bit-level, symmetric, serialization/deserialization implementations for structs and enums
• binrw: binrw helps you write maintainable & easy-to-read declarative binary data readers and writers using ✨macro magic✨.
• scroll: read and write structured data from byte buffers with Pread/Pwrite traits, endian-aware and derive-friendly
• winnow: parser combinator library forked from nom with improved error messages, better developer experience and active maintenance
• antlr-rust-runtime: ANTLR v4 runtime and parser generator with a byte-oriented input stream, so grammars from antlr/grammars-v4 (e.g. bencoding) generate Rust parsers for binary formats. Length-prefixed framing is handled by typed hooks; generation is pure Rust, no JVM

Ruby

• BinData: provides a declarative way to read and write structured binary data

Swift

• swift-binary-parsing (Swift): Apple's official library for building safe, efficient binary parsers with ParserSpan and ParserRange types

Other Programming Languages

• FlexT (Delphi): a DSL and a tool for generating parsers in Delphi
• Haka (Lua): open source security oriented language which allows to describe protocols and apply security policies on (live) captured traffic
• binarylang (Nim): extensible Nim DSL for creating binary parsers/encoders in a symmetric fashion
• binaryparse (Nim): In-language DSL for reading and writing binary data supporting all sorts of patterns. Generates an efficient stream based reader and writer for the runtime execution.
• Gloss (Clojure): turn complicated byte formats into Clojure data structures and Clojure data structures into compact byte representations
• scodec (Scala): Combinator library for working with binary data
• attoparsec and attoparsec-binary: (Haskell): fast parser combinator library, aimed particularly at dealing efficiently with network protocols and complicated text/binary file formats
• Parsifal (OCaml): OCaml-based parsing engine. Paper: A pragmatic solution to the binary parsing problem. Olivier Levillain
• Bindat (Emacs Lisp, builtin): Pack and unpack arrays of bytes, usually for binary network protocols. Unpacking and packing is controlled via a data layout specification, also called Bindat type expression.

Language-Agnostic Tools

Binary Format Description Languages

• Kaitai Struct (DSL): declarative language used for describe various binary data structures, laid out in files or in memory
• RecordFlux: toolset for the formal specification of messages and the generation of verifiable binary parsers and message generators (Ada-inspired).
• Spicy (DSL, C/C++, Zeek): a next-generation parser generator for network protocols and file formats
• DataScript Tools (DSL): DataScript is a formal language for modelling binary datatypes, bitstreams or file formats. PDF
• Dogma (DSL): human-friendly metalanguage for describing data formats in documentation using the familiar patterns of Backus-Naur Form.
• EverParse: a framework for generating verified secure parsers and formatters from domain-specific format specification languages
• ASN.1 (ITU-T/ISO standard): Abstract Syntax Notation One, one of the oldest standards for describing binary data structures with multiple encoding rules (BER, DER, PER, OER). Widely used in telecom, cryptography (X.509), and network protocols (SNMP, LDAP). Tools: asn1c (C compiler), asn1tools (Python)

Standalone Applications

Hex Editors with Grammars

• Synalyze It! (macOS): hex editor with grammar-based binary format parsing
• Hexinator (Windows): hex editor with grammar-based binary format parsing
• 010 Editor (Windows/macOS/Linux): hex editor with C-style binary templates and large template library
• Kiewtai: plugin for the Hiew hex editor that makes the Kaitai parsers available
• Hobbits: multi-platform GUI for bit-based analysis, processing, and visualization. Has a Kaitai plugin.
• ImHex (Windows/macOS/Linux): A Hex Editor for Reverse Engineers, Programmers and people who value their retinas when working at 3 AM.
• Hex Fiend (macOS): fast, open-source hex editor with Tcl-based binary template support for file format parsing
• Okteta (Linux/KDE): hex editor with a Structures tool for analyzing binary data using user-creatable XML or JavaScript structure definitions
• REHex (Windows/macOS/Linux): cross-platform hex editor for reverse engineering with binary templates, annotations, inline disassembly and entropy visualization
• Hex Editor Neo (Windows): hex editor with Structure Viewer, binary templates and Kaitai Struct definition support

Binary Analysis Tools

• GNU poke: The extensible editor for structured binary data
• fq: jq for binary formats - tool, language and decoders for working with binary and text formats
• radare2 (C, with bindings/pipe for almost all languages): Unix-like reverse engineering framework and commandline tools. See Parsing a fileformat with radare2 and Types.
• Veles: open source tool for binary analysis
• PolyFile: identifies and maps the semantic structure of files, including polyglots. Pure-Python reimplementation of libmagic with Kaitai Struct integration and interactive hex viewer

Network Protocol Analysis

• Wireshark: network protocol analyzer that includes dissectors for over two thousand protocols.

  • TShark: command line version, can easily be called from shell scripts.
  • Wireshark Generic Dissector: add-on, allows dissection of a protocol based on a text description of the protocol elements
  • Wireshark Lua: dissectors can be written in Lua (Examples)
  • pyreshark: plugin providing a simple interface for writing Wireshark dissectors in Python
  • Sharktools (Python, Matlab): Tools for programmatic parsing of packet captures using Wireshark functionality

• Netzob: open source tool for reverse engineering, traffic generation and fuzzing of communication protocols

• Cat Karat Packet Builder: packet generation tool that allows to build custom packets for firewall or target testing

• Scapy: send, sniff and dissect and forge network packets.

Research papers

• Interval Parsing Grammars for File Format Parsing (2023): Jialun Zhang, Greg Morrisett, Gang Tan
• LangSec Platform (2021): Towards a Platform to Compare Binary Parser Generators. Olivier Levillain, Sébastien Naud, Aina Toky Rasoamanana (Video)
• Narcissus (2019): Correct-By-Construction Derivation of Decoders and Encoders from Binary Formats. Benjamin Delaware, Sorawit Suriyakarn, Clément Pit-Claudel, Qianchuan Ye, Adam Chlipala
• EverParse (2019): Verified Secure Zero-Copy Parsers for Authenticated Message Formats. Tahina Ramananandro et. al.
• Generic packet descriptions (2017): Verified parsing and pretty printing of low-level data. Marcell van Geest, Wouter Swierstra
• Nail (2014): A Practical Tool for Parsing and Generating Data Formats. Julian Bangert and Nickolai Zeldovich
• FlowSifter (2014): High-Speed Application Protocol Parsing and Extraction for Deep Flow Inspection. Alex X. Liu, Chad R. Meiners, Eric Norige, and Eric Torng
• Zebra (2013): Improving the Performance of Message Parsers for Embedded Systems. Jigar Solanki et. al.
• W. Underwood (2012): Grammar-Based Specification and Parsing of Binary File Formats. William Underwood
• Yakker (2010): Semantics and Algorithms for Data-dependent Grammars. Trevor Jim, Yitzhak Mandelbaum, David Walker
• Zebu (2009): A Language-Based Approach for Improving the Robustness of Network Application Protocol Implementations. Larent Burgy et. al.
• z2z (2009): Automatic Generation of Network Protocol Gateways. Yerom-David Bromberg, Laurent Reveillere, Julia L. Lawall, Gilles Muller
• Tupni (2008): Automatic Reverse Engineering of Input Formats. Weidong Cui et. al.
• PADS/ML (2007): a functional data description language. Y. Mandelbaum, K. Fisher, D. Walker, M. F. Fernandez, and A. Gleyzer.
• BinPAC (2006): Superseded by BinPAC++, which is now known as Spicy
• NetPDL (2006): Markup Language that aims to describe Protocols from OSI layer 2 to OSI layer 7
• TSN.1 (2005): Transfer Syntax Notation One (TSN.1). A formal notation for describing messages in binary protocols
• GAPA (2005): Generic Application-Level Protocol Analyzer and its Language. Nikita Borisov, David J. Brumley, Helen J. Wang, Chuanxiong Guo
• PacketTypes (2000): P. J. McCann and S. Chandra. Packet types: Abstract specification of network protocol messages.

Binary Format References

• Hachoir parser list and source code
• Synalyze It! Grammar Page
• Nom example parsers
• Corkami posters for file formats
• Kaitai Struct format library
• Catalog of FlexT format specifications. A scraper for them. In a e-mail conversation the author of the tool has claimed that the grammars are available under a MIT license. A CoCo/R-like grammar for the DSL also exists, but is in a bit unfinished state.
• Scapy protocols: layers, contrib
• Wikipedia List of file formats

Related Topics

• Language-theoretic Security (LANGSEC)
• Domain Specific Languages
• Digital Forensics, Network Forensics: file format identification, dshell
• Firmware analysis and file carving: Binwalk, Unblob, QuickBMS, OFRAK
• Deep Packet Inspection
• Packet Crafting: hping2/3, tcpreplay, netdude, bittwist, netsniff-ng, Trafgen
• Reverse Engineering
• Fuzzing: Sulley, Peach, ...
• Chomsky hierarchy