Zyte Web Data for Claude Code

From a plain-English prompt to a working Scrapy spider.

---

Not using exclusively Claude Code? See Zyte Coding Agent Add-Ons for alternatives.

Install

claude plugin marketplace add zytedata/claude-skills
claude plugin install zyte-web-data@zyte-ai

If Claude Code is already running, reload plugins in the active session:

/reload-plugins

If /reload-plugins isn't available (e.g. in the VS Code extension), restart Claude Code.

See also: Discovering and installing plugins

---

What it does

This is Zyte's official Claude Code plugin that generates production-ready Scrapy spiders with web-poet page objects from a plain-English prompt. Give it a URL and describe what you want to extract. It handles site exploration, schema discovery, code generation, and smoke testing: no boilerplate, no manual selector hunting.

The plugin explores the target site, discovers available fields, and presents a schema for your approval before generating a single line of code. After you confirm the schema, it creates a Scrapy project with all dependencies configured, generates web-poet page objects and test fixtures, wires up the spider, and runs a smoke test to verify that extraction is working before handing the project back to you.

Optionally, use /scrape-scrapy-cloud to deploy directly to Scrapy Cloud for scheduled runs, job history, and monitoring. A free tier is available.

---

Use cases

The /scrape skill works on any website with repeating structured content: detail pages linked from a listing or category page. Examples from the skill:

• Product catalogs
• Job listings
• Recipes

---

How does it work?

The /scrape skill orchestrates five stages automatically:

1. Decide which fields to extract   →  /scrape-define
2. Analyze the website              →  /scrape-spec
3. Create the Scrapy project        →  /scrape-ensure-project
4. Generate the extraction code     →  /scrape-codegen
5. Generate the spider              →  /scrape-create-spider

Each stage feeds directly into the next. When the pipeline completes, you have a runnable spider and a passing test suite:

uv run scrapy crawl <spider_name>
uv run pytest fixtures/

---

Skills

Orchestration

Pipeline stages (called automatically by /scrape)

Utilities

Deployment

---

Prerequisites

• Claude Code (CLI or desktop app)
• uv — used to create and manage the Scrapy project

Project dependencies (scrapy, scrapy-poet, scrapy-zyte-api, web-poet, extruct, price-parser, pytest) are installed automatically by the skills.

---

Quickstart

Any scraping prompt triggers the skill automatically. For example:

Scrape books.toscrape.com

The plugin walks you through schema approval interactively, then generates a complete, tested Scrapy project.

---

Update

We recommend enabling automatic updates:

1. Enter /plugin in a Claude Code session
2. Select Marketplaces → zyte-ai → Enable auto-update

To update manually:

claude plugin marketplace update zytedata/claude-skills

Then, in a Claude Code session:

/reload-plugins

If /reload-plugins isn't available (e.g. in the VS Code extension), restart Claude Code.

---

Evaluation

We automatically evaluate skills and track both wall time and cost. We measure and aim to improve these metrics over time.

---

Feedback

If you find any issue — such as prompts that did not work as expected, or that caused excessive wall time or cost — please open a GitHub issue.

Provide as much detail as possible to help us reproduce the issue. You are welcome to anonymize target websites or other data.

---

Frequently asked questions

Is a Zyte account required?

No. The generated spider is a standard Scrapy project that runs locally with uv. A Zyte account is required only if you want to deploy to Scrapy Cloud or use Zyte API to access sites that block standard scrapers. If you want to use Zyte API, you'll need an account to generate an API key.

Does it handle JavaScript-rendered pages?

The generated project includes scrapy-zyte-api as a dependency. Enabling headless browser rendering requires a Zyte API key. The /scrape-zyte-login skill guides you through setting up your credentials.

What Python libraries does the generated project use?

The project template includes scrapy, scrapy-poet, scrapy-zyte-api, web-poet, extruct, price-parser, and pytest. All dependencies are installed automatically via uv sync.

Can the generated spider run without Claude Code?

Yes. The plugin generates a standard Scrapy project. Run it directly with:

uv run scrapy crawl <spider_name>

You can extend, modify, and deploy it independently of Claude Code.

---

License

See LICENSE.md for the Zyte End User License Agreement.

---

Demo