Rate Limiting

Websites limit how fast you can hit them.

This is called rate limiting.

Why rate limits exist - protect servers - ensure fair use - stop abuse

If ignored, you may get: - 429 errors - IP bans - blocked accounts

Simple manual delay ```python import time import requests

urls = ["https://example.com/page1", "https://example.com/page2"]

for url in urls: res = requests.get(url) print(res.status_code) time.sleep(2) ```

Token bucket style (concept) You allow N requests per time window.

Using a simple limiter function ```python import time

last_call = 0 def rate_limit(seconds): global last_call now = time.time() wait = seconds - (now - last_call) if wait > 0: time.sleep(wait) last_call = time.time() ```

```python for url in urls: rate_limit(2) requests.get(url) ```

Graph: throttling ```mermaid flowchart LR A[Request] --> B[Wait] B --> C[Next Request] ```

Remember - Always be slower than needed - Respect site limits if documented - Slow scraping is safer scraping