Skip to content
Tutorial Reference

How to Start Web Scraping with Scrapy in Python

Learn how to build scalable web scrapers in Python using Scrapy, covering installation, spider creation, data extraction, and essential settings for production-grade crawling.

When web scraping projects grow beyond simple page fetches, you need a framework that handles concurrency, retries, rate limiting, and data pipelines automatically. Scrapy is Python's premier web scraping framework, designed for extracting data from websites efficiently at scale. Unlike simpler libraries, Scrapy provides an asynchronous architecture, built-in data export, and extensive middleware support: making it the professional choice for production-grade web crawlers.

Installation and Project Setup

Scrapy enforces a structured project layout that organizes spiders, settings, and data pipelines:

pip install scrapy

scrapy startproject bookstore_scraper

cd bookstore_scraper

This generates a project structure:

bookstore_scraper/
├── scrapy.cfg
└── bookstore_scraper/
    ├── __init__.py
    ├── items.py          # Data structure definitions
    ├── middlewares.py    # Request/response processing
    ├── pipelines.py      # Data processing pipelines
    ├── settings.py       # Project configuration
    └── spiders/          # Your spider classes
        └── __init__.py

Creating Your First Spider

Spiders define how to crawl websites and extract data. Create a new file in the spiders/ directory:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"  # Unique identifier for this spider
    
    start_urls = [
        "https://quotes.toscrape.com/page/1/",
    ]
    
    def parse(self, response):
        """Extract data from each page."""
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }
        
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Running the Spider

Execute your spider and export results in various formats:

scrapy crawl quotes -o quotes.json

scrapy crawl quotes -o quotes.csv

scrapy crawl quotes -o quotes.jsonl

Using the Interactive Shell

Test selectors interactively before writing spider code:

scrapy shell "https://quotes.toscrape.com"

In the shell, experiment with selectors:

response.css("div.quote span.text::text").get()
response.css("small.author::text").getall()

response.xpath("//div[@class='quote']//span[@class='text']/text()").get()

Defining Data Structures

For cleaner code and validation, define item classes in items.py:

import scrapy

class QuoteItem(scrapy.Item):
    text = scrapy.Field()
    author = scrapy.Field()
    tags = scrapy.Field()
    scraped_date = scrapy.Field()

Then use them in your spider:

from ..items import QuoteItem
from datetime import datetime

def parse(self, response):
    for quote in response.css("div.quote"):
        item = QuoteItem()
        item["text"] = quote.css("span.text::text").get()
        item["author"] = quote.css("small.author::text").get()
        item["tags"] = quote.css("div.tags a.tag::text").getall()
        item["scraped_date"] = datetime.now().isoformat()
        yield item

Essential Settings

Configure your spider's behavior in settings.py:

ROBOTSTXT_OBEY = True

DOWNLOAD_DELAY = 1

CONCURRENT_REQUESTS = 16

USER_AGENT = "MyBot/1.0 (+http://mywebsite.com/bot)"

AUTOTHROTTLE_ENABLED = True

Scrapy vs. BeautifulSoup

Feature Scrapy BeautifulSoup + Requests
Architecture Asynchronous Synchronous
Speed Very fast (concurrent) Sequential requests
Built-in features Retries, caching, exports Manual implementation
Learning curve Steeper Simpler
Best for Large-scale projects Quick, simple tasks

Quick Command Reference

Command Purpose
scrapy startproject name Create new project
scrapy genspider name domain Generate spider template
scrapy crawl spidername Run a spider
scrapy crawl name -o file.json Run and export data
scrapy shell "url" Interactive testing
scrapy list List available spiders

By adopting Scrapy, you gain a robust framework for building maintainable, scalable web scrapers that handle real-world challenges like rate limiting, retries, and structured data export automatically.

Navigation