Logging and Monitoring

When a scraper runs for hours, print() is not enough.

You need logs.

Why logging matters - know what failed - know when it failed - keep history - debug later

Basic logging setup ```python import logging

logging.basicConfig( filename="scraper.log", level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s" )

logging.info("Scraper started") ```

Log during scraping ```python try: res = requests.get("https://example.com") logging.info("Fetched page successfully") except Exception as e: logging.error("Fetch failed: %s", e) ```

Log progress ```python for i, url in enumerate(urls, 1): logging.info("Processing %d/%d: %s", i, len(urls), url) ```

Graph: logging pipeline ```mermaid flowchart LR A[Scraper] --> B[Logger] B --> C[Log File] C --> D[Review & Debug] ```

Remember - Log start, success, and errors - Use files for long jobs - Logs are your black box recorder