Web Scraping30 min read

Configuration and Secrets

Learn how to manage settings and protect secrets like database URLs and API keys using config files and environment variables.

David Miller
December 21, 2025
0.0k0

Never hardcode: - database passwords - API keys - URLs

They change and must be protected.

---

Use environment variables

```bash export DB_URL="sqlite:///scraped.db" export API_KEY="secret123" ```

```python import os

db_url = os.getenv("DB_URL") api_key = os.getenv("API_KEY") ```

---

Use .env file (with python-dotenv)

``` DB_URL=sqlite:///scraped.db API_KEY=secret123 ```

```python from dotenv import load_dotenv load_dotenv() ```

---

Why config matters - safe secrets - different envs (dev, prod) - easy changes without code edit

---

Graph: config flow

```mermaid flowchart LR A[Env / Config File] --> B[App] B --> C[DB / APIs] ```

---

Remember - Never commit secrets to git - Use env vars for sensitive data - Keep config separate from logic

#Python#Advanced#Config