Security and Safe Scraping

When scrapers move to production, security becomes critical.

You may deal with:

• API keys
• login credentials
• cookies
• private datasets

If leaked, they can cause real damage.

---

Never hard-code secrets

Bad:

API_KEY = "my-secret-key"

Good: use environment variables

import os
API_KEY = os.getenv("API_KEY")

Run with:

export API_KEY="my-secret-key"
python main.py

---

Use .env file (local dev)

API_KEY=my-secret-key
DB_PASS=strongpassword

from dotenv import load_dotenv
load_dotenv()

---

Validate external input

Never trust scraped or user data blindly.

def clean(text):
    return text.strip()[:200]

---

Avoid logging secrets

Do NOT log:

• passwords
• tokens
• cookies

---

Graph: secure flow

flowchart LR
  A[Secrets] --> B[Env Variables]
  B --> C[Scraper]
  C --> D[Safe Usage]

---

Remember

• Secrets go in env vars
• Sanitize all data
• Think like an attacker