scrapy-impersonate

scrapy-impersonate is a Scrapy download handler. This project integrates curl_cffi to perform HTTP requests, so it can impersonate browsers' TLS signatures or JA3 fingerprints.

Installation

pip install scrapy-impersonate

Activation

To use this package, replace the default http and https Download Handlers by updating the DOWNLOAD_HANDLERS setting:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

By setting USER_AGENT = None, curl_cffi will automatically choose the appropriate User-Agent based on the impersonated browser:

USER_AGENT = ""

Also, be sure to install the asyncio-based Twisted reactor for proper asynchronous execution:

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Usage

Set the impersonate Request.meta key to download a request using curl_cffi:

import scrapy


class ImpersonateSpider(scrapy.Spider):
    name = "impersonate_spider"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "USER_AGENT": "",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_impersonate.RandomBrowserMiddleware": 1000,
        },
    }

    def start_requests(self):
        for _ in range(5):
            yield scrapy.Request(
                "https://tls.browserleaks.com/json",
                dont_filter=True,
            )

    def parse(self, response):
        # ja3_hash: 98cc085d47985d3cca9ec1415bbbf0d1 (chrome133a)
        # ja3_hash: 2d692a4485ca2f5f2b10ecb2d2909ad3 (firefox133)
        # ja3_hash: c11ab92a9db8107e2a0b0486f35b80b9 (chrome124)
        # ja3_hash: 773906b0efdefa24a7f2b8eb6985bf37 (safari15_5)
        # ja3_hash: cd08e31494f9531f560d64c695473da9 (chrome99_android)

        yield {"ja3_hash": response.json()["ja3_hash"]}

impersonate-args

You can pass any necessary arguments to curl_cffi through impersonate_args. For example:

yield scrapy.Request(
    "https://tls.browserleaks.com/json",
    dont_filter=True,
    meta={
        "impersonate": browser,
        "impersonate_args": {
            "verify": False,
            "timeout": 10,
        },
    },
)

Some arguments worth knowing about:

[!WARNING] allow_redirects is forced to False so that redirects are handled by Scrapy. Re-enabling it through impersonate_args makes curl_cffi follow redirects internally, which bypasses Scrapy's redirect and offsite middlewares and exposes you to redirect-based SSRF.

curl-options

For settings that have no curl_cffi argument, impersonate_curl_options passes raw libcurl options through. Keys can be CurlOpt members or their names, and they take precedence over the options set by this handler.

The most common use is pinning the header order, which browsers keep stable and WAFs check (HTTPHEADER_ORDER requires curl_cffi >= 0.16.0):

yield scrapy.Request(
    "https://tls.browserleaks.com/json",
    dont_filter=True,
    meta={
        "impersonate": "chrome",
        "impersonate_curl_options": {
            "HTTPHEADER_ORDER": "Host,Connection,User-Agent,Accept,Referer",
        },
    },
)

Supported browsers

The following browsers can be impersonated (curl_cffi >= 0.15.0):

_Notes:

1. The old Safari target names (safari15_3, safari15_5, safari17_0, safari17_2_ios, safari18_0, safari18_0_ios) are kept as deprecated aliases. Prefer the new names (safari153, safari155, …) for new code.
2. You can also pass a floating value like impersonate="chrome", "safari", "safari_ios" or "firefox" to let curl_cffi pick the most recent fingerprint without pinning a version.
3. RandomBrowserMiddleware rotates across chrome, firefox, safari, edge and tor by default. Override with IMPERSONATE_BROWSERS to narrow the set (e.g. IMPERSONATE_BROWSERS = ["chrome", "firefox"]).

Development

Install the development dependencies and run the test suite:

pip install -r requirements-dev.txt
pip install -e .
pytest

The tests spin up local HTTP/HTTPS servers and a CONNECT proxy, so no network access is required.

Thanks

This project is inspired by the following projects:

• curl_cffi - Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
• curl-impersonate - A special build of curl that can impersonate Chrome & Firefox
• scrapy-playwright - Playwright integration for Scrapy