Maintenance and Long-Term Running

Real scrapers live long. They need care.

Challenges:

• HTML changes
• network failures
• new blocks
• growing data

---

Detect site change

if not items:
    logging.warning("No items found, possible site change")

---

Add alerts (simple)

if errors > 5:
    send_email("Scraper failing!")

---

Backup data

Always backup DB or files before overwriting.

---

Graceful retry

import time

for i in range(3):
    try:
        html = fetch(url)
        break
    except Exception:
        time.sleep(5)

---

Graph: long-term cycle

flowchart LR
  A[Run] --> B[Store]
  B --> C[Monitor]
  C --> D[Fix]
  D --> A

---

Remember

• Expect failures
• Monitor results
• Maintenance is part of scraping