Maintained by Zyte with over 500 other contributors
The world’s most-used open source data extraction framework
Crawl sites, extract structured data, and ship it anywhere. Lean by design, extensible by nature — trusted by professionals extracting public web data at scale.
The tutorial spider, crawling quotes.toscrape.com
Extensions
A lean core. Everything else is a plugin away.
Scrapy handles the crawling. Battle-tested extensions from the community and Zyte add browser rendering, monitoring, anti-ban, and now direct access for AI agents.
MCP lets AI agents like Claude connect directly to your tools. This server gives them live access to running crawls — list jobs, check status, and execute Python inside the crawl process without leaving your agent.
$claude mcp add --scope user scrapy-mcp -- uvx --from scrapy-mcp-official scrapy-mcp
Render JavaScript-heavy pages in a real browser while keeping the request/response workflow you already use.
Learn moreValidate scraped data and get alerted on Slack, Discord, or email the moment a spider breaks.
Learn moreAutomatic proxy rotation, browser fingerprinting, and ban avoidance through Zyte API — no infrastructure to run.
Learn moreSeparate extraction from crawling so each piece can be tested and reused on its own.
Learn moreHow it works
Lean by design
Six small parts around one asynchronous engine, every one of them pluggable. Pick a part to see the code — and what it produces.
Step 03 · Spider
Spiders are the part you write. They receive responses, pull data out with CSS or XPath selectors, yield items, and follow links to discover more pages.
Extend it: Generic spiders like CrawlSpider and SitemapSpider cover the common crawl patterns for you.
Read the docsimport scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com/"]
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small::text").get(),
}
yield from response.follow_all(
css="li.next a", callback=self.parse
){"text": "The world as we have created it is aprocess of our thinking.", "author": "Albert Einstein"}{"text": "It is our choices, Harry, that show what wetruly are.", "author": "J.K. Rowling"}[engine] DEBUG: Crawled (200) <GET …/page/2/>{"text": "This life is what you make it.","author": "Marilyn Monroe"}
Try selectors before you write the spider
The interactive shell loads any page so you can test CSS and XPath live. Shell docs
$scrapy shell "https://quotes.toscrape.com"
The Engine drives all six asynchronously. See the full architecture
Workflow
From first spider to production
The same project that runs on your laptop scales to millions of pages — with tooling for every step along the way.
Build
Scaffold a project in one command — or hand a URL to an AI agent with Scrapy Agent Skills and get a tested spider back.
$scrapy startproject quotes
Run & debug
Try selectors in the interactive shell, then inspect live crawls straight from your agent with the Scrapy MCP Server.
$scrapy shell "https://quotes.toscrape.com"
Deploy & monitor
Push to Scrapy Cloud, schedule runs, and let spidermon alert you the moment your data drifts.
$shub deploy
Actively maintained
Fifteen years of releases, and counting
Regular releases from a team of maintainers and 500+ contributors, with a deprecation policy that keeps upgrades boring.
- v2.19.0LatestSeptember 2026
- New RemoteControl extension which allows inspecting and controlling a running crawl over HTTP, used by the Scrapy MCP server
- Experimental aiohttp-based download handler (now the default when running without a reactor)
- v2.18.0August 2026
- HttpxDownloadHandler now uses httpx2
- The Twisted-based HTTP/2 download handler is no longer experimental
- v2.17.0July 2026
- Security bug fixes
- HTTP/2 and SOCKS proxy support for HttpxDownloadHandler
Trusted by developers in production
“If it wasn't for Scrapy, my freelancing career, and then the scraping business would have never taken off. The Scrapy framework, and especially its documentation, simplifies crawling and scraping for anyone with basic Python skills. I don't know, now there is this emotive bond with Scrapy that I've developed over the years.”
Nishant Choudhary
Founder of DataFlirt.com
“Scrapy: the gift that revolutionised web scraping and inspired me to pay it forward with Scrapoxy!"
"Scrapy is the central engine of all my projects, since its 1st release.”
Fabien Vauchelles
Creator of Scrapoxy
“Scrapy stands out as an exceptional tool for seamlessly managing open requests, making large-scale data collection efficient and reliable."
"With its remarkable scraping speed, Scrapy transforms complex web extraction tasks into swift, streamlined operations that save time and maximise output.”
Hyder Khan
Flipdish
“Scrapy is the cornerstone of web scraping with Python. Without it, scraping would be much harder."
"During my career, I've built thousands of scrapers, mostly with Scrapy. It made my life a lot easier.”
Pierluigi Vinciguerra
Co-Founder and CTO at Databoutique.com
“With over 6 years in web scraping, Scrapy has been my go-to framework for building fast, reliable, and scalable data pipelines."
"In my 6+ years of scraping experience, nothing has matched Scrapy’s balance of power, flexibility, and community support.”
Ganesh Satheendran
Turbolab Technologies
“When Google Search enforced JavaScript, our scraper broke. Struggling to build a working alternative with Puppeteer on a short notice, we fired up Scrapy, Splash, and whipped up a script. Lo and behold – it worked! Our Google benchmark was buzzing again.”
Adam
Proxyway
Want to get involved?
Support
Where to find help
Fifteen years of docs, answers, and a community that has probably hit your problem before.
The tutorial, API reference, and topic guides — start here.
OpenAsk questions, share what you're building, and talk to maintainers.
OpenGuides, blog posts, and tutorials from the community and Zyte.
BrowseReport bugs, browse good first issues, and contribute code.
OpenStart with one spider. Scale to millions of pages.
Free, open source, and running in production for over fifteen years.