Web Scraping30 min read

Configuration and Secrets

Learn how to manage settings and protect secrets like database URLs and API keys using config files and environment variables.

David Miller
December 17, 2025
1.4k41

Never hardcode:

  • database passwords
  • API keys
  • URLs

They change and must be protected.


Use environment variables

export DB_URL="sqlite:///scraped.db"
export API_KEY="secret123"
import os

db_url = os.getenv("DB_URL")
api_key = os.getenv("API_KEY")

Use .env file (with python-dotenv)

DB_URL=sqlite:///scraped.db
API_KEY=secret123
from dotenv import load_dotenv
load_dotenv()

Why config matters

  • safe secrets
  • different envs (dev, prod)
  • easy changes without code edit

Graph: config flow

flowchart LR
  A[Env / Config File] --> B[App]
  B --> C[DB / APIs]

Remember

  • Never commit secrets to git
  • Use env vars for sensitive data
  • Keep config separate from logic
#Python#Advanced#Config