Reddit Data Downloader and Thread Reconstructor

This command-line tool downloads Reddit submissions and comments, reconstructs comment trees, and exports the result as CSV, JSON, or both. It does not require a database.

It uses one of two sources depending on the requested dates:

• Before 1 January 2026 (UTC): selected .zst archive files are downloaded from the supplied Academic Torrents torrent with libtorrent.
• From 1 January 2026 (UTC): submissions and comments are retrieved from the Arctic Shift API and saved locally as JSONL files.

The tool can combine both sources when a date range crosses the boundary.

Prerequisites

• Windows 10/11 (the commands below are for Windows; the tool also runs on macOS and Linux).
• Internet access. Torrent downloads also need peers to be available.
• Miniconda or Anaconda. Miniconda is the smaller download and is sufficient.

The project is set up for Python 3.9. Use a 64-bit Python/Conda installation on a typical 64-bit Windows PC.

Windows setup (Conda + Python 3.9)

Already have a working 64-bit Python 3.9 environment? You may skip the Miniconda installation and Conda environment-creation steps, activate your existing Python 3.9 virtual environment instead, and continue at Step 4. Any other method of running Python 3.9 is also fine. Do not use a newer Python version: this project's libtorrent dependency is intended to run with Python 3.9. If you are unsure which Python version or environment you have, follow the Miniconda steps below.

1. Install Miniconda

1. Download the Windows x86_64 installer from the Miniconda installation page.
2. Run the installer and complete the default installation.
3. Open Anaconda Prompt from the Start menu. Do not use a regular Command Prompt unless Conda has already been initialized there.

2. Open this project folder

In Anaconda Prompt, run:

cd "C:\reddit download script\reddit_data_downloader_tool"

If you saved the project elsewhere, replace that path with its actual location.

3. Create and activate an isolated Python 3.9 environment

conda create --name reddit-downloader python=3.9 pip -y
conda activate reddit-downloader

You should now see (reddit-downloader) at the beginning of the prompt. Activate this environment again whenever you want to run the tool.

4. Install libtorrent and the Windows DLL helper

Install libtorrent first, then the package that provides its required OpenSSL DLLs on Windows:

python -m pip install --upgrade pip
python -m pip install libtorrent
python -m pip install libtorrent-windows-dll

The second package prevents the common ImportError: DLL load failed while importing libtorrent error. It installs libcrypto-1_1-x64.dll and libssl-1_1-x64.dll alongside the libtorrent package.

Verify that libtorrent imports successfully:

python -c "import libtorrent as lt; print('libtorrent version:', lt.version)"

If this command prints a version number, continue. If it reports a DLL error, confirm that you are using the activated reddit-downloader environment, then rerun the two libtorrent install commands above.

5. Install the remaining project packages

python -m pip install -r requirements.txt

The requirements file also includes the Windows DLL helper, so this command is safe to rerun later to repair or update the environment.

Run the downloader

From the project directory, with the environment active, start the program:

conda activate reddit-downloader
python downloader.py

Answer the prompts as follows:

1. Subreddit name(s): Enter one subreddit name or several comma-separated names. Do not include r/.

   Fiverr, Upwork
   

2. Start date: Enter YYYY-MM-DD, for example 2025-01-01.

3. End date: Enter YYYY-MM-DD, for example 2025-02-01. The end date is an exclusive boundary, so this example requests data from 1 January up to, but not including, 1 February.

4. Export format: choose one of the following:

   • 1 - CSV: flat rows for posts and comments, including available raw fields.
   • 2 - JSON: posts with nested comment and reply trees.
   • 3 - both CSV and JSON.

The program reports which source it will use, downloads the required data, reads it into memory, and writes the export files when finished. Keep the terminal open until it displays EXPORT COMPLETE.

Important notes about data coverage and speed

• If your requested interval includes any pre-2026 date, the tool downloads the full available pre-2026 archive file(s) for each requested subreddit from Academic Torrents. It does not download only the historical portion of that interval.
• For 2026 and later, the Arctic Shift API request is limited to the interval you enter. Downloading can be slow because it relies on a community-operated API. Please be patient and avoid repeatedly rerunning large requests; the slower pace helps prevent abuse of a valuable community resource.
• The data was manually spot-checked against the Reddit website during development and appeared suitable for analytics, research, category comparisons, and tracking trends over time. It is not guaranteed to be complete or exact, so do not rely on it where precise totals or complete coverage are required.

Data sources

• Historical archives: Academic Torrents Reddit archive, believed to have been made by u/Watchful1.
• Recent-data API: Arctic Shift.

Downloading historical archives with Academic Torrents

Any requested date before 2026-01-01 UTC needs the reddit-archives.torrent file in the project folder.

1. Check whether reddit-archives.torrent is already beside downloader.py. This project currently includes one.

2. If it is missing, download the appropriate Reddit archive .torrent file from Academic Torrents.

3. Rename the downloaded file exactly to reddit-archives.torrent.

4. Put it in the project root:

   C:\reddit download script\reddit_data_downloader_tool\reddit-archives.torrent
   

5. Run python downloader.py and enter the historical date range. If the file is missing, the program pauses and asks you to add it before continuing.

The torrent downloader searches the torrent metadata for files named <subreddit>_submissions.zst and <subreddit>_comments.zst, then prioritizes only those matching files. Download speed and availability depend on the torrent's peers.

Background Seeding Support

The tool includes native support for seeding downloaded .zst files back to the Academic Torrents network to keep them available and fast for other researchers.

• Seeding Prompt: When requesting pre-2026 data (or if you already have .zst files in your downloads/ folder), downloader.py will ask if you would like to run a background seeder.
• Non-intrusive Execution: If you accept, a separate windowless background process (seeder.pyw) is spawned using pythonw.exe on Windows (which has no visible command window) or standard background threads on macOS/Linux.
• Controls: You can set the maximum upload speed limit in Megabits per second (Mbps) and set a duration limit (e.g., 6h for 6 hours, 2d for 2 days).
• Automatic Single Instance: The seeder script creates an OS-level lock (seeder.lock). If the downloader runs and detects that a seeder is already active in the background, it will skip asking and avoid launching duplicate background processes.
• Logs: The seeder logs its progress and status silently in seeder.log.

Files created

• downloads\ - cached source files: .zst torrent archives and/or .jsonl API responses.
• reddit_export_YYYYMMDD_HHMMSS.csv - CSV export, when you choose CSV or both.
• reddit_export_YYYYMMDD_HHMMSS.json - reconstructed nested JSON export, when you choose JSON or both.

Exports are written to the project folder. Cached downloads are retained, so remove them manually only if you no longer need them.

Roadmap and feedback

More data categories and optional PRAW support may be added in the future.

If you try the tool, feedback is welcome. In particular, please report whether it works in your environment, whether the installation instructions are clear, and any errors you encounter. Open an issue or send a message with the error details, your operating system, Python version, and the command you ran so the problem can be reproduced and fixed.

Quick troubleshooting