Websites inspect headers to decide who you are.

If no User-Agent:
They think you are a bot.

## What is User-Agent
It tells:
- browser
- OS
- device

Example:
Mozilla/5.0 (Windows NT 10.0; Win64; x64)...

## Always send headers
```python
headers = {
  "User-Agent": "Mozilla/5.0"
}

res = requests.get("https://example.com", headers=headers)
```

## Request flow
```mermaid
flowchart LR
  A[Scraper] -->|Headers| B[Server]
  B --> C{Looks Real?}
  C -->|Yes| D[Send Page]
  C -->|No| E[Block]
```

## Remember
- User-Agent is mandatory
- Mimic a real browser


Understand HTTP headers, especially User-Agent, and why many sites block requests without them.

User-Agent and Headers

Learn web scraping with Python, BeautifulSoup, Selenium, and advanced scraping techniques.

User-Agent and Headers

What is User-Agent It tells: - browser - OS - device

Always send headers ```python headers = { "User-Agent": "Mozilla/5.0" }

Request flow ```mermaid flowchart LR A[Scraper] -->|Headers| B[Server] B --> C{Looks Real?} C -->|Yes| D[Send Page] C -->|No| E[Block] ```

Remember - User-Agent is mandatory - Mimic a real browser

More on Web Scraping

Web Scraping Intro

How Websites Work

HTTP Requests Basics