Arbitrage Data

This is the repository of data associated with the Flash Boys 2.0: Frontrunning, Transaction Reordering, and Consensus Instability in Decentralized Exchanges paper.

Various Scripts

As you can see, there are various python scripts in the root of this repository. These scripts take the raw transaction data from the Etherum network as stored in a SQL DB and parse it to find information about gas auctions.

• calculate_profit_from_logs.py - calculates arbitrage profits from solidity log events. Uses Google BigQuery dataset.
• calculate_slots.py - calculate the price slots for gas auctions
• count_wins.py - count the number of times each arbitrager won an auction
• csv_hack.py - cleans data for parsing
• csv_to_sqlite.py - collates data from multiple CSVs into one database
• exchanges.py - scrapes and parses data from a list of well known distributed exchanges
• filter_list.txt - list of addresses to ignore
• gastoken.py - script to identify if an arbitrager is using GasToken
• generate_graphs.py - Generates various graphs
• get_all_arb_receipts.py - Gets Ethereum transaction reciepts for successful arbitrageurs.
• get_auction_slots_intersection.py - Gets reciepts for bidders to auctions.
• get_bq_blocks.py - Gets block data from Google BigQuery and puts it in a CSV.
• get_bq_fees.py - Gets block fee data from Google BigQuery and puts it in a CSV.
• get_bq_logs.py - Gets emitted logs and other transaction data from Google BigQuery and puts it in a CSV.
• get_bq_relayers.py - Gets emitted logs from the addresses for bancor, kyber and uniswap.
• get_bq_summarystats.py - Collates summary statistics from the BigQuery-scraped CSVs.
• get_bq_txlist.py - Gets various transaction data from Google BigQuery.
• get_pairwise_data.py - Pull out pairs of players in an auction from the auctions CSV.
• persistence.py - Helper function.
• read_csv.py - Creates the auctions CSV from the raw collected data from the go-ethereum monitoring software.
• receipts.py - Helper function to write reciepts.
• scrape_gasauctions.py - scrapes auctions from a full node by requesting block data.
• sqlite_adapter.py - Helper function to query SQLite.
• update.py - Helper script to automate updating the dataset as time progresses.
• write_csv.py - Connects to the SQL DB and retrieves the data from the SQL database and writes it out to a CSV for parsing.

data

Contains a list of known relayers for various exchanges.

etherdelta

Contains scripts to perform scraping of Etherdelta's transaction order book data.

go-ethereum

This directory contains the source code for our fork of go-ethereum that we developed to monitor the ethereum network and collect transaction data as it propogated accross the network. Primarily the files that are of interest are:

• go-ethereum/eth/arb_monitor.go
• go-ethereum/eth/MonitorListGetter
• go-ethereum/eth/handler.go
• arbmonmon/

graph_templates

LaTeX source code for generating graphs (used for the paper).

paper

The LaTeX source of the associated paper.

webapp

The web application source code for the auction monitoring dashboard, which displays the data for monitoring of gas auctions as they happen.