Logging and Monitoring

When a scraper runs for hours, print() is not enough.

You need logs.

Why logging matters

• know what failed
• know when it failed
• keep history
• debug later

Basic logging setup

import logging

logging.basicConfig(
    filename="scraper.log",
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s"
)

logging.info("Scraper started")

Log during scraping

try:
    res = requests.get("https://example.com")
    logging.info("Fetched page successfully")
except Exception as e:
    logging.error("Fetch failed: %s", e)

Log progress

for i, url in enumerate(urls, 1):
    logging.info("Processing %d/%d: %s", i, len(urls), url)

Graph: logging pipeline

flowchart LR
  A[Scraper] --> B[Logger]
  B --> C[Log File]
  C --> D[Review & Debug]

Remember

• Log start, success, and errors
• Use files for long jobs
• Logs are your black box recorder