The world’s most-used open source data extraction framework
A collaborative, open source framework for extracting public web data.
Fast & Powerful
Write the rules to extract the data you need, and let Scrapy do the rest.
Customizable
Build spiders in Python and tailor them to any site or data model.
Open Source
Maintained by a thriving community, used by millions of developers worldwide.
Deploy in minutes
Build spiders faster
Select a feature to see the corresponding code snippet.
- Create a Scrapy Project
- Initialize a new Scrapy project with a single command.
- Run a Spider
- Deploy a Scrapy spider to scrape data.
- Export Data
- Save extracted data to a file in your format of choice.
- Use Scrapy Shell
- Test and debug your scraping logic interactively.
- Monitor Crawls
- Track the performance of your spiders.
$ scrapy startproject myprojectWhy Create a Scrapy Project?
Creating a Scrapy project sets up the necessary folder structure and files to start building your web scraper efficiently.
Learn more about Why Create a Scrapy Project?Join the community
“ If it wasn't for Scrapy, my freelancing career, and then the scraping business would have never taken off. The Scrapy framework, and especially its documentation, simplifies crawling and scraping for anyone with basic Python skills. I don't know, now there is this emotive bond with Scrapy that I've developed over the years.”
Nishant Choudhary
Founder of DataFlirt.com
“ Scrapy: the gift that revolutionised web scraping and inspired me to pay it forward with Scrapoxy!"
"Scrapy is the central engine of all my projects, since its 1st release.”
Fabien Vauchelles
Creator of Scrapoxy
“ Scrapy stands out as an exceptional tool for seamlessly managing open requests, making large-scale data collection efficient and reliable."
"With its remarkable scraping speed, Scrapy transforms complex web extraction tasks into swift, streamlined operations that save time and maximise output.”
Hyder Khan
Flipdish
“ Scrapy is the cornerstone of web scraping with Python. Without it, scraping would be much harder."
"During my career, I've built thousands of scrapers, mostly with Scrapy. It made my life a lot easier.”
Pierluigi Vinciguerra
Co Founder and CTO at Databoutique.com
“ With over 6 years in web scraping, Scrapy has been my go-to framework for building fast, reliable, and scalable data pipelines."
"In my 6+ years of scraping experience, nothing has matched Scrapy’s balance of power, flexibility, and community support.”
Ganesh Satheendran
Turbolab Technologies
“ When Google Search enforced JavaScript, our scraper broke. Struggling to build a working alternative with Puppeteer on a short notice, we fired up Scrapy, Splash, and whipped up a script. Lo and behold – it worked! Our Google benchmark was buzzing again.”
Adam
Proxyway
Join millions of developers using Scrapy.
55.1k Stars, 10.8k Forks.
Fast, free web scraping backed by a thriving community.