A Scrapy spider needs to scrape a site fronted by **Cloudflare Turnstile**. This recipe solves the token via CaptchaSonic and submits the protected form so the spider can crawl the gated pages.

> [!NOTE]
> Turnstile tokens are single-use and tied to the IP that requested them. Pass a `proxy=` so the SDK solves through the same egress IP your Scrapy spider uses — otherwise Cloudflare may reject the token.

---

## What you'll build

A Scrapy spider that posts a Turnstile-protected form, receives the cookies that gate the rest of the site, and crawls the protected pages. ~10 minutes.

---

## Setup

```bash
pip install scrapy captchasonic
export CAPTCHASONIC_API_KEY=sonic_xxx
```

You need the page's Turnstile **sitekey** (in `<div class="cf-turnstile" data-sitekey="…">`) and the URL of the form it gates.

---

## The recipe

```python
import os
import scrapy
from captchasonic import CaptchaSonic

class GatedSpider(scrapy.Spider):
    name = "gated"
    start_urls = ["https://example.com/protected-form"]

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.solver = CaptchaSonic(os.environ["CAPTCHASONIC_API_KEY"])

    def parse(self, response):
        sitekey = response.css(".cf-turnstile::attr(data-sitekey)").get()
        if not sitekey:
            self.logger.error("no turnstile widget on %s", response.url)
            return

        # 1. Solve the token (auto-polls; ~6–12 s typical).
        result = self.solver.solve_turnstile(
            website_url=response.url,
            website_key=sitekey,
            # Solve through the same proxy your spider uses — token must
            # be IP-coherent for Cloudflare to accept it.
            proxy=os.environ.get("SPIDER_PROXY"),
        )
        token = result["token"]

        # 2. Submit the form with the token attached.
        yield scrapy.FormRequest.from_response(
            response,
            formdata={"cf-turnstile-response": token},
            callback=self.after_pass,
        )

    def after_pass(self, response):
        # Cookies set on this response will be reused for the rest of the crawl.
        for href in response.css("a.item::attr(href)").getall():
            yield response.follow(href, self.parse_item)

    def parse_item(self, response):
        yield {
            "url": response.url,
            "title": response.css("h1::text").get(),
        }
```

> Returns a Turnstile token that you submit as the form's `cf-turnstile-response`. Once the form accepts it, Cloudflare's `cf_clearance` cookie is set and Scrapy reuses it on every following request in the same session.

---

## Common pitfalls

- **`cf_clearance` cookie not set after form submit.** Cloudflare requires the token-bearing request to come from the *same IP and User-Agent* that solved it. In `settings.py`: pin `USER_AGENT` and route both the SDK call (`proxy=`) and Scrapy (`HTTPPROXY_ENABLED`) through the same proxy.
- **Hitting the rate limit.** `MinuteLimitExceededError` carries a `retry_after` (seconds). Hook it into `DownloaderMiddleware.process_response` to back off cleanly.
- **Solving for every page.** You only need to solve **once** per session — the form's response sets `cf_clearance`, which Scrapy will carry on subsequent requests via its cookie jar.

---

## See also

- [Python SDK — Cloudflare Turnstile](/docs/clients-sdks/python#cloudflare-turnstile)
- [Capability Matrix](/docs/clients-sdks/capabilities) — token field for `solve_turnstile` is `result["token"]`
- [Configuration](/docs/clients-sdks/python#configuration) — polling/timeout tuning for headless crawlers
