lorien/awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

JavaScript

8,173

568 commits

updated Sep 19, 2026

See the code

README

Awesome Web Scraping

List of packages, services, and manuals related to web scraping.

Topics

Python - Python packages
PHP - PHP packages
Ruby - Ruby packages
JavaScript - JavaScript packages
Go - Go packages
Command Line Tools - tools with a command line interface
Web-Scraping Manuals - list of articles and books teaching web scraping
dhamaniasad / HeadlessBrowsers - list of (almost) all headless web browsers in existence
DNS over HTTPS providers - list of DNS over HTTPS providers
Awesome Pastebins - list of pastebin sites
Awesome Tunneling - List of tunneling software and services. Focus on self-hosting.

Captcha Solving Services

https://2captcha.com

Proxy Server Marketplaces

Telegram Discussion Groups

@grablab - talks in English
@grablab_ru - talks in Russian

How to Contribute to This List

Please see the CONTRIBUTING.md document.

Credits

The list is initially based on partial data from these sources awesome-python, awesome-php, awesome-ruby, ruby-nlp, awesome-javascript

captcha-bypass

captcha-recaptcha

crawler

crawling

crawling-framework

crawling-python

crawling-tool

scraping

scraping-framework

scraping-python

scraping-tool

spider

web-scraping

webscraping

Project health

Overall

3.4/10

Maintained

10/10

Code review

0/10

OpenSSF Scorecard

Significant stargazers

Bryan C Guner

339 followers · starred Aug 2022

Chirag Singhal

97 followers · starred Nov 2021

Paul Keen

125 followers · starred Aug 2015

Heinz-Alexander Fuetterer

47 followers · starred Jan 2025

Similar repos

BruceDone/awesome-crawler

A collection of awesome web crawler,spider in different languages

7,328

D4Vinci/Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a…

86,889

niespodd/browser-fingerprinting

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system…

5,155

transitive-bullshit/awesome-puppeteer

A curated list of awesome puppeteer resources.

2,584

apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers.…

9,584

apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In…

26,126

gocolly/colly

Elegant Scraper and Crawler Framework for Golang

25,550

vavkamil/awesome-bugbounty-tools

A curated list of various bug bounty tools

6,321