Project Structure and Clean Code

When scraping becomes serious, one-file scripts are not enough.

A clean structure helps you:

• debug faster
• reuse code
• add new scrapers easily
• work in teams

---

Recommended folder layout

scraper/
│
├── main.py
├── fetcher.py
├── parser.py
├── storage.py
├── config.py
├── utils.py
├── requirements.txt
└── logs/

Meaning:

• fetcher: HTTP requests
• parser: HTML → data
• storage: DB logic
• config: URLs, settings
• utils: helpers

---

Example: separation of concerns

fetcher.py

import requests

def fetch(url):
    return requests.get(url, timeout=10).text

parser.py

from bs4 import BeautifulSoup

def parse(html):
    soup = BeautifulSoup(html, "html.parser")
    return [h.text for h in soup.select("h2.title")]

main.py

from fetcher import fetch
from parser import parse

html = fetch("https://example.com")
items = parse(html)
print(items)

---

Graph: clean flow

flowchart LR
  A[main.py] --> B[fetcher]
  B --> C[parser]
  C --> D[storage]

---

Remember

• One file, one responsibility
• Clean structure saves time later
• This is how production scrapers are organized