Useful extensions

Plenty of modern sites render their content with JavaScript. A plain Scrapy request only sees the empty HTML shell — the data you actually need never shows up.

scrapy-playwright drives a real browser to render the page, then hands you the fully-loaded HTML — while keeping the same request/response workflow you already use in Scrapy.

# settings.py
DOWNLOAD_HANDLERS = {
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

# spider.py
yield scrapy.Request(url, meta={"playwright": True})

Spiders fail quietly in production. A broken selector, a redirected URL, or a site redesign can return empty or wrong data for days before anyone notices.

spidermon adds monitoring, validation, and alerting — Slack, Discord, or email — to your spiders, so you find out the moment something breaks, not a week later.

# settings.py
EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}
SPIDERMON_ENABLED = True
SPIDERMON_SPIDER_CLOSE_MONITORS = (
    "myproject.monitors.SpiderCloseMonitorSuite",
)

Sites fight back with IP bans, browser fingerprinting, and CAPTCHAs. Hand-rolling proxy rotation and retry logic eats up development time — and still breaks.

scrapy-zyte-api connects your spiders to Zyte's managed API for automatic proxy rotation, browser fingerprinting, and ban avoidance, with no custom infrastructure to run.

# settings.py
ADDONS = {
    "scrapy_zyte_api.Addon": 500,
}
ZYTE_API_KEY = "<your API key>"

When extraction logic is tangled up with crawling logic, spiders get hard to test, reuse, and maintain — every HTML tweak on the target site means hunting through the whole spider.

scrapy-poet separates extraction logic from parsing logic with page objects, so each piece can be tested and reused on its own. Pair it with Web Scraping Copilot (VS Code Extension) to generate page objects automatically.

# page.py
class ProductPage(WebPage):
    def to_item(self) -> dict:
        return {
            "name": self.css("h1::text").get(),
            "price": self.css(".price::text").get(),
        }