The world’s most-used open source data extraction framework
Lean by design, extensible by nature — trusted by professionals extracting public web data at scale.
Agent Skills for Scrapy
Created by Zyte
Turn a URL into a production-ready spider — from Claude Code, GitHub Copilot, or any AI agent.
How it works
A project built to scale
Scrapy scaffolds a full project, not a throwaway script — keeping your spiders, settings, items, and pipelines organized as your crawler grows. Run scrapy startproject to lay it out, then scrapy crawl to run.
quotes/ # project root├── scrapy.cfg # deploy configuration└── quotes/ # the project's Python module├── __init__.py├── items.py # define the data you extract├── middlewares.py├── pipelines.py # clean, validate & store items├── settings.py # project-wide configuration└── spiders/ # your spiders live here├── __init__.py└── quotes.py # one spider per source
Ecosystem
Extend Scrapy with community add-ons
Scrapy stays lean by design. A growing ecosystem of community and Zyte extensions adds browser rendering, monitoring, anti-ban, and more — only when you need it.
Lean by design
Everything you need in core — everything else is an extension away
Select a capability to see it in code.
- Powerful selectors
- Extract with CSS or XPath — mix them freely.
- Crawl at scale
- Async engine with smart, polite throttling.
- Item pipelines
- Validate, clean, and store every item.
- Export anywhere
- Feed exports to JSON, CSV, S3 and more.
- Interactive shell
- Test selectors live before writing code.
- Extensible by design
- Hook in middlewares, signals and extensions.
# CSS or XPath — your choice, even mixed
title = response.css("h1::text").get()
price = response.xpath("//p[@class='price_color']/text()").get()
# Get everything, with a sensible default
tags = response.css("div.tags a.tag::text").getall()
image = response.css("img::attr(src)").get(default="")
# Run regex straight off a selector
sku = response.css("p.sku::text").re_first(r"SKU:\s*(\w+)")Selectors
Scrapy's built-in selectors let you query the DOM with CSS or XPath, extract attributes, set defaults, and even apply regex — all from one chainable API.
Learn more about SelectorsActively maintained
Over a decade of releases, and counting
Scrapy has been in continuous development for over 15 years, with regular releases from a team of maintainers and 500+ contributors.
HTTP/2 and SOCKS proxy support, plus improved TLS version settings.
Official Python 3.14 support and Twisted 26.4.0+ compatibility.
Experimental reactor-less mode and a new httpx-based download handler.
Trusted by developers in production
“ If it wasn't for Scrapy, my freelancing career, and then the scraping business would have never taken off. The Scrapy framework, and especially its documentation, simplifies crawling and scraping for anyone with basic Python skills. I don't know, now there is this emotive bond with Scrapy that I've developed over the years.”
Nishant Choudhary
Founder of DataFlirt.com
“ Scrapy: the gift that revolutionised web scraping and inspired me to pay it forward with Scrapoxy!"
"Scrapy is the central engine of all my projects, since its 1st release.”
Fabien Vauchelles
Creator of Scrapoxy
“ Scrapy stands out as an exceptional tool for seamlessly managing open requests, making large-scale data collection efficient and reliable."
"With its remarkable scraping speed, Scrapy transforms complex web extraction tasks into swift, streamlined operations that save time and maximise output.”
Hyder Khan
Flipdish
“ Scrapy is the cornerstone of web scraping with Python. Without it, scraping would be much harder."
"During my career, I've built thousands of scrapers, mostly with Scrapy. It made my life a lot easier.”
Pierluigi Vinciguerra
Co Founder and CTO at Databoutique.com
“ With over 6 years in web scraping, Scrapy has been my go-to framework for building fast, reliable, and scalable data pipelines."
"In my 6+ years of scraping experience, nothing has matched Scrapy’s balance of power, flexibility, and community support.”
Ganesh Satheendran
Turbolab Technologies
“ When Google Search enforced JavaScript, our scraper broke. Struggling to build a working alternative with Puppeteer on a short notice, we fired up Scrapy, Splash, and whipped up a script. Lo and behold – it worked! Our Google benchmark was buzzing again.”
Adam
Proxyway
Want to get involved?
Join millions of developers using Scrapy.
Fast, free web scraping backed by a thriving community.