Building a Production Web Scraper: Why I Chose Scrapy

Today, both side projects built for learning and actual production systems scrape the internet to retrieve data and populate systems with valuable insights. Whether it's aggregating product listings, building datasets for machine learning, or monitoring changes at scale, web crawling is a fundamental engineering skill — and the choices you make about how to crawl matter more than most people think.

I scraped thousands of recipes and ingredient encyclopedias from a popular recipe website for my food AI platform — a project I started for fun that's becoming quite interesting. I started with requests + BeautifulSoup, rewrote everything in Scrapy, and ran an 84-test migration audit to prove the new scrapers produce identical data. This is what I learned about engineering choices, testing, and when a framework actually matters.

The Starting Point: Two Scripts That Worked

My original setup was two standalone Python scripts — crawler.py (335 lines) and ingredient_crawler.py (548 lines). They used requests.Session() with a spoofed Chrome User-Agent, manual exponential backoff retries, hardcoded time.sleep() delays, and mixed extraction logic with database persistence.

They worked. They populated my database with 848 recipes and 230 ingredients. But they had problems that only surfaced over time — bugs in category classification, arbitrary page limits, no robots.txt respect, and a spooky SynchronousOnlyOperation error I'd eventually hit when trying to scale.

Why Scrapy and Not Something Else

I evaluated three options before committing to Scrapy:

Option 1: Stay with requests + BeautifulSoup

What I already had. Zero migration cost, but every improvement (concurrency, retries, robots.txt, rate limiting) requires writing more ad-hoc code that's already been written better in Scrapy.

Option 2: Use httpx + parsel (or playwright)

A lighter-weight alternative. httpx gives you async HTTP, parsel gives you CSS/XPath selectors (same engine Scrapy uses). But you still build everything else yourself: the scheduler, the dupefilter, the middleware chain, the item pipeline, the spider lifecycle. You end up building a worse Scrapy.

Option 3: Scrapy

The batteries-included framework. It gives you robots.txt checking, retry middleware, throttling, concurrent requests, deduplication, item pipelines, and a shell for interactive debugging. The tradeoff is a learning curve and opinionated architecture — but those opinions are exactly what prevent the bugs I had.

I chose Scrapy because concurrency, error handling, rate limiting, and robots.txt compliance are problems that must be solved one way or another — and doing them manually means reimplementing what a battle-tested framework already handles better. The things that actually required my domain knowledge — extraction logic, HTML parsing strategies, category classification — I keep identical, just moved into spider methods instead of standalone functions.

Engineering Decisions That Mattered

1. ROBOTSTXT_OBEY = True and an Honest User-Agent

My old crawler spoofed a Chrome User-Agent and never checked robots.txt. The target site's robots.txt blocks AI crawlers entirely (GPTBot, ClaudeBot, etc.) and disallows paths like /admin/ and /search/. I was accidentally compliant only because I happened to target allowed paths.

# settings.py
ROBOTSTXT_OBEY = True
USER_AGENT = 'RecipeBot (+https://github.com/recipebot)'

This is an ethical and practical decision. An honest User-Agent means the site operator can contact me, and robots.txt compliance means I don't accidentally crawl pages I shouldn't. If the site decides to block my bot, I'll know immediately instead of silently bypassing their rules.

2. No Arbitrary Page Limits

My old crawler had MAX_PAGES = 15 hardcoded. When a recipe category had 20 pages, I silently missed the last 5. Scrapy spiders follow next-page links until they stop existing:

def parse_category(self, response, category):
    # Extract recipe links, then follow next page
    for link in response.css('article.recipe-card a[href*="recipes"]::attr(href)').getall():
        yield response.follow(link, callback=self.parse_recipe, cb_kwargs={'category': category})

    next_page = response.css('a[rel="next"]::attr(href)').get()
    if next_page:
        yield response.follow(next_page, callback=self.parse_category, cb_kwargs={'category': category})

No MAX_PAGES. The site itself tells me when I'm done. I currently get ~224 recipes per category; the old crawler was capped at whatever 15 pages yielded.

3. xpath('string()') Instead of ::text

This was a subtle bug I caught during migration testing. Scrapy's ::text pseudo-selector only gets direct text nodes, missing text inside nested elements like <a><strong>Pasta</strong> 00</a>. My old BeautifulSoup.get_text() grabbed all text including nested elements.

# WRONG: misses text inside <strong>, <em>, etc.
name = dd.css('a::text').get()        # returns "00" not "Pasta 00"

# CORRECT: gets all text including nested elements
name = dd.css('a').xpath('string()').get('')  # returns "Pasta 00"

I normalize with ' '.join(text.split()) to match the old clean_text() behavior. This was caught by my comparison tests — the old crawler produced "Farina 00 300 g" while the new one initially produced "Farina 00 300\n\t\t\t\t\t\t\t\t\t\tg".

4. Image URL Normalization

The ingredient encyclopedia serves images from the main domain, but Scrapy's response.urljoin() resolves relative URLs against the current domain. My old crawler explicitly normalized to the main domain:

# Old crawler: force all URLs to the main domain
if src.startswith('//'):
    src = 'https:' + src
elif src.startswith('/'):
    src = 'https://www.example.com' + src

I replicate this in the Scrapy spider as a _normalize_url() static method. Without it, og:image paths resolved to the encyclopedia subdomain instead of the main site, breaking image downloads.

5. Category Rule Ordering Matters

My classify_ingredient() function uses substring matching. This means rule order is critical:

• "mela" (Frutta keyword) matches "Melanzane" (a vegetable)
• "fagiol" (Verdure keyword for green beans) matches "Fagioli" (a legume)
• "pisell" (Verdure keyword for peas) matches "Piselli" (a legume)

The fix: place Legumi before Verdure in the rule list so fagioli and piselli match the legume rules first, and place Verdure before Frutta so melanzan matches before mela does:

CATEGORY_RULES = [
    ('Carni',      ['agnell', 'manzo', ...]),
    ('Pesce',      ['salmone', 'tonno', ..., 'rana pescatrice']),
    ('Latticini',  ['mozzarell', 'parmigian', ...]),
    ('Cereali e Pasta', ['pasta', 'spaghetti', ...]),
    ('Erbe e Aromi', ['basilico', 'rosmarino', ...]),
    ('Spezie',     ['pepe', 'cannella', ...]),
    ('Legumi',     ['fagioli', 'lenticchi', ...]),  # BEFORE Verdure
    ('Verdure',    ['carota', 'melanzan', ...]),     # BEFORE Frutta
    ('Frutta',     ['mela', 'pera', ...]),
    ('Condimenti', ['olio', 'aceto', ...]),
    ('Bevande',    ['vino', 'birra', ...]),
    ('Dolci',      ['cioccolato', 'cacao', ...]),
]

This is the kind of bug that unit testing catches before it becomes a data problem.

6. Separation of Concerns: Spiders vs. Pipelines

My old crawler.py did everything in one function: fetch the URL, parse the HTML, extract the data, download the image, save to the database, track progress. Scrapy enforces a structure where:

• Spiders only extract data and yield Item objects
• Pipelines receive items and handle persistence
• Settings configure cross-cutting concerns (retries, throttling, robots.txt)

# Spider: extraction only
yield RecipeItem(
    title=title, url=response.url, ingredienti=ingredienti,
    steps=steps, category=category, ...
)

# Pipeline: persistence only
class RecipePipeline:
    def process_item(self, item, spider):
        self.Recipe.objects.update_or_create(url=item['url'], defaults={...})
        return item

This means changing the database schema doesn't touch the spider. Changing the extraction logic doesn't touch the pipeline. You can test each independently.

The Category Bug: A Case Study in Substring Matching

I discovered "rana pescatrice" (monkfish) classified as Frutta in production. The root cause was rule ordering: "mela" in the Frutta keyword list matched as a substring of "Melanzane" (eggplant), and "Melanzane" was falling into Frutta before the Verdure rule could match "melanzan". Similarly, "fagiol" (green beans, Verdure) matched as a prefix of "Fagioli" (legume).

These bugs existed in the old crawler too — they weren't introduced by Scrapy. The difference is that now I have 44 unit tests that catch them:

class ClassifyIngredientTestCase(TestCase):
    def test_fish_classification(self):
        self.assertEqual(classify_ingredient('Rana pescatrice'), 'Pesce')  # was Frutta!
        self.assertEqual(classify_ingredient('Coda di rospo'), 'Pesce')
    
    def test_vegetable_classification(self):
        self.assertEqual(classify_ingredient('Melanzane'), 'Verdure')  # was Frutta!
        self.assertEqual(classify_ingredient('Pomodori'), 'Verdure')
    
    def test_legumes_classification(self):
        self.assertEqual(classify_ingredient('Fagioli'), 'Legumi')  # was Verdure!
        self.assertEqual(classify_ingredient('Piselli'), 'Legumi')   # was Verdure!

robots.txt: What I Learned

The target site's robots.txt:

User-agent: *
Disallow: /ingredients/
Disallow: /admin/
Disallow: /search/
Disallow: /recipes/*?

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

My old crawler: never checked it, spoofed a Chrome User-Agent, and hoped for the best. I never hit disallowed paths by accident, but only by coincidence.

With Scrapy's ROBOTSTXT_OBEY = True, every URL is checked against the site's published rules before being fetched. If I add a new spider that targets a disallowed path, Scrapy will refuse to crawl it and log why.

Testing: What I Have and What I Need

What I Have: 42 Unit Tests

My test suite covers three layers:

Layer 1: Unit tests for helper functions — classify_ingredient(), truncate_at_sentence(), match_section(). These are pure functions with no network or database dependencies:

class ClassifyIngredientTestCase(TestCase):
    def test_fish_classification(self):
        self.assertEqual(classify_ingredient('Rana pescatrice'), 'Pesce')
        self.assertEqual(classify_ingredient('Coda di rospo'), 'Pesce')
    
    def test_vegetable_classification(self):
        self.assertEqual(classify_ingredient('Melanzane'), 'Verdure')  # not Frutta!
        self.assertEqual(classify_ingredient('Patata'), 'Verdure')
    
    def test_unknown_classification(self):
        self.assertEqual(classify_ingredient('Acqua'), 'Altro')

Layer 2: Spider parser tests — Feed mock HTML to spider parse methods and assert the output. No network required:

class RecipeSpiderParseTestCase(TestCase):
    def test_parse_recipe_basic(self):
        spider = RecipeSpider()
        json_ld = json.dumps({"@type": "Recipe", "estimatedCost": "Basso",
                               "prepTime": "PT20M", "cookTime": "PT15M",
                               "recipeYield": 4, "image": "https://example.com/img.jpg"})
        html = f'<html><body><h1>Pasta al Pomodoro</h1>...<script type="application/ld+json">{json_ld}</script>...</body></html>'
        response = HtmlResponse(url='https://...', body=html.encode('utf-8'))
        results = list(spider.parse_recipe(response, category='Primi'))
        self.assertEqual(len(results), 1)
        self.assertEqual(results[0]['title'], 'Pasta al Pomodoro')
        self.assertEqual(results[0]['difficulty'], 'Facile')
        self.assertEqual(results[0]['servings'], 4)

    def test_parse_recipe_no_title_returns_none(self):
        # Spider should skip pages with no title
        ...
    
    def test_ingredient_text_with_nested_elements(self):
        # xpath('string()') vs ::text — catches the nested <strong> bug
        ...

Layer 3: Migration comparison tests — I ran an 84-test audit where I fetched the same pages with both the old and new crawlers and compared every field. This was a one-time manual test, but it proved the Scrapy spiders produce identical output:

RECIPE COMPARISON: 20/20 passed (all fields match)
INGREDIENT COMPARISON: 10/10 passed (all fields match)
DB CONSISTENCY: 10/10 passed (spider output matches stored records)
CATEGORY CLASSIFICATION: 44/44 passed

What I Need: Regression Tests for Continuous Crawling

If the scraper runs as a daily/weekly job, I need automated regression testing. Here's what the ideal setup looks like:

1. Frozen fixture tests. Save representative HTML pages to tests/fixtures/ and commit them. Tests parse these fixtures instead of hitting the live site. This catches regressions when I change spider code, and decouples CI from network availability:

# tests/test_recipe_spider.py
FIXTURE_DIR = Path(__file__).parent / 'fixtures'

class RecipeSpiderRegressionTest(TestCase):
    def test_category_listing(self):
        with open(FIXTURE_DIR / 'category_page1.html') as f:
            response = HtmlResponse(url='...', body=f.read().encode('utf-8'))
        spider = RecipeSpider()
        results = list(spider.parse_category(response, category='Antipasti'))
        self.assertTrue(len(results) > 0)
        for result in results:
            self.assertIn('recipes', result.url)

    def test_recipe_detail(self):
        with open(FIXTURE_DIR / 'recipe_detail.html') as f:
            response = HtmlResponse(url='...', body=f.read().encode('utf-8'))
        spider = RecipeSpider()
        results = list(spider.parse_recipe(response, category='Primi'))
        self.assertEqual(results[0]['title'], 'Risotto allo zafferano')
        self.assertEqual(len(results[0]['ingredienti']), 10)
        self.assertEqual(results[0]['difficulty'], 'Media')

2. Contract tests. Assert that every RecipeItem and IngredientInfoItem yielded by a spider has all required fields populated:

class RecipeItemContractTest(TestCase):
    def test_all_fields_populated(self):
        spider = RecipeSpider()
        html = open(FIXTURE_DIR / 'recipe_detail.html').read()
        response = HtmlResponse(url='...', body=html.encode('utf-8'))
        item = list(spider.parse_recipe(response, category='Primi'))[0]
        
        required_fields = ['title', 'url', 'image_url', 'ingredienti', 
                          'steps', 'category', 'difficulty', 'prep_time',
                          'cook_time', 'servings']
        for field in required_fields:
            self.assertIn(field, item)
            self.assertIsNotNone(item[field])

3. Schema change detector. If the target site changes their HTML (e.g., they rename dd.recipe-ingredient to dd.ingredient-list), my selectors break silently — I'd still get items, but with empty ingredient lists. A schema test catches this:

class RecipeSchemaTest(TestCase):
    def test_selectors_still_valid(self):
        """Fetch a live page and verify selectors find expected elements."""
        # This test hits the real site — run it weekly, not on every commit
        spider = RecipeSpider()
        response = fetch_live('https://example.com/risotto.html')
        item = list(spider.parse_recipe(response, category='Primi'))[0]
        
        self.assertTrue(len(item['ingredienti']) >= 3, 
            f"Expected >=3 ingredients, got {len(item['ingredienti'])}. "
            "Selectors may have broken.")
        self.assertTrue(len(item['steps']) >= 2,
            f"Expected >=2 steps, got {len(item['steps'])}. "
            "Selectors may have broken.")

4. Category classification snapshot. Store a list of known ingredient→category mappings and assert they don't change unexpectedly:

KNOWN_CLASSIFICATIONS = {
    'Salmone': 'Pesce',
    'Pollo': 'Carni',
    'Melanzane': 'Verdure',  # Was Frutta before the bug fix
    'Fagioli': 'Legumi',     # Was Verdure before the bug fix
    'Rana pescatrice': 'Pesce',  # Was Carni before the bug fix
}

class CategoryRegressionTest(TestCase):
    def test_known_classifications_stable(self):
        for name, expected in KNOWN_CLASSIFICATIONS.items():
            result = classify_ingredient(name)
            self.assertEqual(result, expected,
                f"classify_ingredient('{name}') returned '{result}', expected '{expected}'")

This test prevents a keyword change from silently regressing "Rana pescatrice" back to Frutta. Every time I add a new keyword, I add it to KNOWN_CLASSIFICATIONS.

5. Pipeline integration test. Feed a known RecipeItem through the Django pipeline and verify it creates/updates the database correctly:

class RecipePipelineTest(TestCase):
    def test_update_or_create(self):
        pipeline = RecipePipeline()
        item = RecipeItem(
            title='Test Recipe',
            url='https://example.com/test',
            category='Primi',
            ingredienti=['Pasta 400g'],
            steps=['Boil water'],
            difficulty='Facile',
            prep_time='10 min',
            cook_time='15 min',
            servings=4,
            image_url='',
        )
        pipeline.process_item(item, None)
        
        recipe = Recipe.objects.get(url='https://example.com/test')
        self.assertEqual(recipe.title, 'Test Recipe')
        self.assertEqual(recipe.ingredienti, ['Pasta 400g'])
        
        # Run again — should update, not duplicate
        item['title'] = 'Updated Test Recipe'
        pipeline.process_item(item, None)
        self.assertEqual(Recipe.objects.filter(url='https://example.com/test').count(), 1)
        recipe.refresh_from_db()
        self.assertEqual(recipe.title, 'Updated Test Recipe')

The Final Architecture

scrapy_spider/
  settings.py          # ROBOTSTXT_OBEY, AUTOTHROTTLE, honest User-Agent, DJANGO_ALLOW_ASYNC_UNSAFE
  items.py             # RecipeItem, IngredientInfoItem — data contracts
  pipelines.py         # RecipePipeline, DjangoIngredientPipeline, classify_ingredient()
  spiders/
    recipes.py         # RecipeSpider — extraction only, no DB calls
    ingredients.py     # IngredientSpider — extraction only, section-based parsing
run_crawler.py          # Entry point
recipes/tests.py        # 42 unit tests (classification, parsing, helpers, contracts)

Run it:

python3 run_crawler.py recipes       # Crawl all recipe categories
python3 run_crawler.py ingredients   # Crawl ingredient encyclopedia
python3 run_crawler.py all           # Crawl everything
python3 manage.py test recipes.tests # Run 42 unit tests

What I Gained

Key Takeaways

1. Use a framework for production scraping. requests + BeautifulSoup is fine for 10 pages. For 848 recipes and 230 ingredients across paginated categories, Scrapy's architecture pays for itself on day one.
2. Respect robots.txt. It's not optional. ROBOTSTXT_OBEY = True and an honest User-Agent prevent accidental abuse and make you a good internet citizen.
3. Don't hardcode limits. Crawl until the site stops giving you pages. My MAX_PAGES = 15 was silently dropping data.
4. xpath('string()') over ::text. If your source HTML has nested elements (<strong>, <em>, <span> inside <a> or <li>), ::text silently drops text. xpath('string()') gets everything. I caught this during migration testing.
5. Rule ordering matters for substring classifiers. "mela" matches "Melanzane". "fagiol" matches "Fagioli". Place your rules in specificity order and write regression tests.
6. Test spiders with fixtures, not live sites. Commit representative HTML pages. Test parsing against those. Reserve live-site tests for weekly CI, not every commit.
7. Separate extraction from persistence. Spiders parse HTML and yield items. Pipelines save to databases. This means you can test extraction without a database, and swap databases without touching the spider.
8. Add contract tests. Assert that every item has all required fields. If the target site changes dd.recipe-ingredient to dd.ingredient-list, you'll get empty ingredient lists. A contract test catches this before it reaches production.