wmd-straw

This is wmd-straw (or wayback-machine-downloader-straw), a fork of the Wayback Machine Downloader. With this, you can download a website from the Internet Archive Wayback Machine.

Included here is partial content from other forks, namely those from ShiftaDeband and matthid — attributions are in the code and go to the original authors — as well as a few additional features.

Quick start

Download a website's latest snapshot:

wayback_machine_downloader https://example.com

Your files will be saved to ./websites/example.com/ with their original structure preserved.

Installation

Requirements

• Ruby 3+ (download Ruby here)
• Bundler gem (gem install bundler)

Quick install

It took a while, but we have a gem for this! Install it with:

gem install wayback_machine_downloader_straw

To run most commands, just like in the original WMD, you can use:

wayback_machine_downloader https://example.com

Note that you can also manually download this repository and run commands locally by appending ruby before a command (e.g., ruby wayback_machine_downloader https://example.com).

Conflict warning: This gem may conflict with hartator's original wayback_machine_downloader gem. You might need to uninstall the original for this fork to work. A good way to tell is if a command fails and lists the gem version as 2.3.1 or earlier; this WMD fork uses 2.3.2 or above.

Native executable releases

Oh hey, this is a thing now. If you go to the Actions tab, you'll see a workflow that builds native executables for Linux, macOS, and Windows whenever a new version tag is pushed. The resulting binaries are uploaded to the GitHub Release for that version. Note: this is experimental!

Step-by-step setup

1. Install Ruby:

   ruby -v
   

   This will verify your installation. If it's not installed, download Ruby for your OS.

2. Install dependencies:

   bundle install
   

   If you encounter an error like cannot load such file -- concurrent-ruby, manually install the missing gem:

   gem install concurrent-ruby
   

3. Run it:

   cd path/to/wayback-machine-downloader/bin
   ruby wayback_machine_downloader https://example.com
   

   For example, if you extracted the contents to a folder named "wayback-machine-downloader" in your Downloads directory, you'd type cd Downloads\wayback-machine-downloader\bin.

Windows tip: In File Explorer, Shift + Right Click your bin folder → "Open Terminal here".

Docker users

We have a Docker image! See Packages for the latest version. You can also build it yourself:

docker build -t wayback_machine_downloader .
docker run -it --rm wayback_machine_downloader [options] URL

As an example of how this works without cloning this repo, this command fetches smallrockets.com until the year 2013:

docker run -v .:/build/websites ghcr.io/strawberrymaster/wayback-machine-downloader:master wayback_machine_downloader --to 20130101 smallrockets.com

Using Docker Compose

You can also use Docker Compose, which makes it easier to extend functionality (such as implementing a database to store previous downloads):

# docker-compose.yml
services:
  wayback_machine_downloader:
    build:
        context: .
    tty: true
    image: wayback_machine_downloader:latest
    container_name: wayback_machine_downloader
    volumes:
      - .:/build:rw
      - ./websites:/build/websites:rw

Usage

Create the image named "wayback_machine_downloader":

docker compose up -d --build

Run the container:

docker compose run --rm wayback_machine_downloader https://example.com [options]

Configuration

There are a few constants in wayback_machine_downloader.rb that can be edited for your convenience. The default values may be conservative, so you can adjust them to your needs:

DEFAULT_TIMEOUT = 30        # HTTP timeout (in seconds)
MAX_RETRIES = 3             # Number of times to retry failed requests
RETRY_DELAY = 2             # Wait time between retries (seconds)
RATE_LIMIT = 0.25           # Throttle between requests (seconds)
CONNECTION_POOL_SIZE = 10   # Maximum simultaneous connections
MEMORY_BUFFER_SIZE = 16384  # Download buffer size (bytes)
STATE_CDX_FILENAME = '.cdx.json'       # Stores snapshot listing
STATE_DB_FILENAME = '.downloaded.txt'  # Tracks completed downloads

Advanced usage

Basic options

Download to specific directory

ruby wayback_machine_downloader https://example.com --directory downloaded-backup/

By default, files are downloaded to ./websites/ followed by the domain name. Use this to save files elsewhere.

Download historical timestamps

ruby wayback_machine_downloader https://example.com --all-timestamps 

This downloads all snapshots for a website, using the timestamp as the directory name. For example:

websites/example.com/20060715085250/index.html
websites/example.com/20051120005053/index.html

Download content from a specific timeframe

# On or after July 16, 2006
ruby wayback_machine_downloader https://example.com --from 20060716231334 

# On or before September 16, 2010
ruby wayback_machine_downloader https://example.com --to 20100916231334

Timestamps can be found inside the URLs of the regular Wayback Machine website. You can also use years (2006) or year + month (200607). These flags can be combined to create a range.

Download exact URL

ruby wayback_machine_downloader https://example.com --exact-url

Retrieves only the file matching exactly the URL provided, avoiding the rest of the site.

Download rewritten files

ruby wayback_machine_downloader https://example.com --rewritten

Useful if you want to download the files exactly as they exist on the Wayback Machine (with their rewriting injected) instead of the original raw files.

Recursive subdomains with local links

ruby wayback_machine_downloader https://example.com --recursive-subdomains --subdomain-depth 1 --local

Grabs the site and any discovered subdomains one level deep, rewriting links to work locally.

Rewrite links to work locally without downloading (EXPERIMENTAL)

ruby wayback_machine_downloader --local-only example.com

Useful if you already have the files downloaded and just want to rewrite the links to work locally without running the download process again. This will not download any files, only rewrite links in existing HTML/CSS/JS files. If you want to have these files in a separate directory, make sure to point it to the correct directory (e.g., wayback_machine_downloader --local-only example.com --directory /path/to/custom/dir), or the folder you save your downloaded files in.

---

Page requisites

Download pages plus referenced assets

wayback_machine_downloader https://example.com --page-requisites

Useful if WMD didn't naturally pick up all the content you wanted.

---

Composite snapshot at a point in time

Freeze the site as of Jan 1, 2013

wayback_machine_downloader https://example.com --snapshot-at 20130101000000

---

Filtering content

Include only specific files

# Only images
ruby wayback_machine_downloader https://example.com -o "/\.(jpg|png)/i"

# Only files in a specific folder
ruby wayback_machine_downloader https://example.com --only my_directory

You can supply a string or a regex (using the /regex/ notation).

Exclude specific files

# Everything except images
ruby wayback_machine_downloader https://example.com --exclude "/\.(gif|jpg|jpeg)$/i"

---

Performance

Increase concurrency

ruby wayback_machine_downloader https://example.com --concurrency 20

May significantly speed up the download by fetching multiple files at once.

Increase snapshot limit

ruby wayback_machine_downloader https://example.com --maximum-snapshot 300    

100 pages is the default maximum and is sufficient for most websites. Use a bigger number for massive sites.

---

Diagnostics

Download all files (including errors)

ruby wayback_machine_downloader https://example.com --all

By default, the downloader limits itself to successful responses and follows redirect captures to their final content. This flag adds error files (40x, 50x) and saves redirect pages (30x) themselves, as well as empty files that are usually removed.

Keep duplicate captures (disable digest collapse)

ruby wayback_machine_downloader https://example.com --keep-duplicates

Useful when you need separate captures even if they have identical response bodies, such as some redirect source URLs.

Generate URL list

ruby wayback_machine_downloader https://example.com --list

Displays a JSON list of files to be downloaded with timestamps and URLs. Useful for debugging or piping to another application.

---

Job management

The downloader automatically saves its progress (.cdx.json for snapshot list, .downloaded.txt for completed files) in the output directory. If you run the same command again pointing to the same output directory, it will resume where it left off.

[!NOTE] Automatic resumption can be affected by changing the URL, mode selection (like --all-timestamps), or filtering options. If you want to ensure a clean start, use the --reset option.

Restart a download job

ruby wayback_machine_downloader https://example.com --reset

Useful if you suspect the state files are corrupted or want a fresh process without deleting your downloaded files.

Keep state files

ruby wayback_machine_downloader https://example.com --keep

Useful for debugging or if you plan to extend the download later with different parameters (e.g., adding a --to timestamp).

---

Troubleshooting

SSL certificate errors

If you encounter an SSL error like:

SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get certificate CRL)

This is a known issue with OpenSSL 3.6.0 on certain Ruby installations, not a bug with this WMD fork specifically. (See ruby/openssl#949).

The workaround is to create a file named fix_ssl_store.rb with the following content:

require "openssl"
store = OpenSSL::X509::Store.new.tap(&:set_default_paths)
OpenSSL::SSL::SSLContext::DEFAULT_PARAMS[:cert_store] = store

Then run the downloader using:

RUBYOPT="-r./fix_ssl_store.rb" wayback_machine_downloader "http://example.com"

Verifying the issue: You can test if your Ruby environment has this issue by running this snippet. If it fails with the same SSL error, the workaround above will fix it:

require "net/http"
require "uri"

uri = URI("https://web.archive.org/")
Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
  resp = http.get("/")
  puts "GET / => #{resp.code}"
end

---

Contributing

1. Fork the repository
2. Create a feature branch
3. Submit a pull request

Run tests (note, these are still broken!):

bundle exec rake test