Master web scraping from basics to advanced techniques. Learn BeautifulSoup, Selenium, handling dynamic content, and building robust scrapers.

Learn what web scraping is, how it works, HTTP basics, HTML structure, and ethical scraping practices.

Web Scraping Fundamentals

Master BeautifulSoup for parsing HTML, finding elements, extracting data, and handling common patterns.

BeautifulSoup & Basic Scraping

Handle JavaScript-rendered content with Selenium, manage sessions, cookies, and authentication.

Advanced Scraping Techniques

Store scraped data in databases, CSV files, JSON. Clean and process data for analysis.

Data Storage & Processing

Build robust scrapers with error handling, rate limiting, proxies, and anti-bot bypass techniques.

Production-Ready Scrapers

Web Scraping with Python

Learn web scraping with Python, BeautifulSoup, Selenium, and advanced scraping techniques.

Web Scraping

Web scraping means:  
  **automatically collecting data from websites using code instead of copying by hand.**
  
  If a website shows:
  - product prices
  - news articles
  - job listings
  - sports scores  
  
  and you want that data in your program or database, you use web scraping.
  
  ## Why web scraping exists
  Before scraping:
  - people copied data manually
  - very slow and error-prone
  
  With scraping:
  - computers fetch pages
  - extract only what you need
  - store it automatically
  
  This saves hours or days of work.
  
  ## Simple real examples
  - Track product prices daily
  - Collect jobs from job portals
  - Monitor news headlines
  - Gather data for research
  - Build datasets for AI models
  
  ## When did it become popular
  Web scraping started becoming common:
  - in early 2000s with search engines
  - grew fast with Python libraries like BeautifulSoup
  - exploded with data science and AI after 2015
  
  Today, scraping is a core skill for:
  - data engineers
  - analysts
  - backend developers
  - researchers
  
  ## How scraping works (big picture)
  1) Request a web page (like a browser)
  2) Get HTML code
  3) Find elements you need
  4) Extract text/links/images
  5) Save data
  
  ## Graph: scraping flow
  ```mermaid
  flowchart LR
    A[Python Script] --> B[Send HTTP Request]
    B --> C[Website Server]
    C --> D[HTML Response]
    D --> E[Parse HTML]
    E --> F[Extract Data]
    F --> G[Save to File/DB]
  ```
  
  ## Is scraping legal?
  It depends:
  - Public data is usually allowed
  - Respect robots.txt
  - Do not overload servers
  - Do not scrape private/logged-in data
  - Follow site terms
  
  Always scrape **responsibly**.
  
  ## Why use Python for scraping
  Python is popular because:
  - simple syntax
  - powerful libraries
  - huge community
  - easy data processing
  
  ## Future of web scraping
  Scraping is still growing because:
  - more data-driven systems
  - AI needs large datasets
  - businesses depend on live data
  
  But:
  - sites are adding anti-bot systems
  - scrapers must become smarter
  - APIs will replace some scraping
  
  So learning scraping now is still very valuable.
  
  ## Remember
  - Scraping = automatic data collection from web
  - Saves time and builds datasets
  - Python is the best beginner choice
  - Always scrape ethically
  

A complete beginner-friendly introduction to web scraping: what it is, why it exists, how it works, its history, benefits, risks, and future.

Web Scraping Intro

Before scraping, you must understand how a website works.
  
  A website is not magic.  
  It is just:
  - a server
  - sending text (HTML)
  - to your browser.
  
  Your scraper does the same.
  
  ## What happens when you open a site
  1) You enter a URL
  2) Browser sends request
  3) Server responds with HTML
  4) Browser renders page
  
  Your scraper will stop at step 3 and read HTML.
  
  ## What is HTML
  HTML is a text document with tags:
  
  ```html
  <h1>News</h1>
  <p>This is a paragraph</p>
  <a href="/jobs">Jobs</a>
  ```
  
  Tags describe structure, not data meaning.
  
  ## Key parts for scraping
  - Tags: h1, p, div, span, a
  - Attributes: class, id, href
  - Text inside tags
  
  ## Static vs Dynamic websites
  ### Static
  HTML already contains data.
  Easy to scrape.
  
  ### Dynamic
  HTML loads empty, data comes later via JavaScript.
  Harder to scrape, needs browser automation.
  
  ## Graph: static vs dynamic
  ```mermaid
  flowchart TD
    A[Request Page] --> B{Type?}
    B -->|Static| C[HTML has data]
    B -->|Dynamic| D[JS loads data later]
  ```
  
  ## Developer tools (your best friend)
  In browser:
  - Right click → Inspect
  - See HTML structure
  - Find tags and classes
  
  This is how you decide what to scrape.
  
  ## Remember
  - Scraper reads HTML, not visuals
  - Learn to inspect elements
  - Static sites are easier
  - Dynamic sites need extra tools
  

Understand how websites send HTML to your browser so you know what exactly your scraper is downloading and reading.

How Websites Work

Scraping starts with sending HTTP requests.
  
  HTTP is the language computers use to talk on the web.
  
  ## What is a request
  A request asks:
  “Please give me this page.”
  
  A response says:
  “Here is the HTML.”
  
  ## First request with Python
  ```python
  import requests
  
  url = "https://example.com"
  res = requests.get(url)
  
  print(res.status_code)
  print(res.text[:200])
  ```
  
  ## Status codes (important)
  - 200: OK
  - 404: Not found
  - 403: Forbidden
  - 500: Server error
  
  Always check status before scraping.
  
  ## Headers: act like a browser
  Some sites block unknown bots.
  
  ```python
  headers = {
    "User-Agent": "Mozilla/5.0"
  }
  res = requests.get(url, headers=headers)
  ```
  
  ## Why headers matter
  They tell server:
  - who you are
  - what you accept
  - how to respond
  
  ## Graph: request cycle
  ```mermaid
  flowchart LR
    A[Python] --> B[HTTP GET]
    B --> C[Server]
    C --> D[Status + HTML]
    D --> A
  ```
  
  ## Common mistakes
  - Ignoring status codes
  - No User-Agent
  - Sending too many requests fast
  
  ## Remember
  - requests library is your base tool
  - Always check status
  - Use headers to mimic browser
  

Learn how your scraper talks to websites using HTTP requests, status codes, headers, and why they matter.

HTTP Requests Basics

After downloading HTML, you must parse it.
  
  BeautifulSoup helps you:
  - read HTML
  - navigate tags
  - extract text and attributes
  
  ## Install
  ```bash
  pip install beautifulsoup4
  ```
  
  ## Basic usage
  ```python
  from bs4 import BeautifulSoup
  
  html = "<h1>Title</h1><p>Text</p>"
  soup = BeautifulSoup(html, "html.parser")
  
  print(soup.h1.text)
  print(soup.p.text)
  ```
  
  ## Parse real page
  ```python
  import requests
  from bs4 import BeautifulSoup
  
  res = requests.get("https://example.com")
  soup = BeautifulSoup(res.text, "html.parser")
  ```
  
  ## Find elements
  ```python
  soup.find("h1")
  soup.find_all("p")
  soup.find("div", class_="news")
  ```
  
  ## Extract attributes
  ```python
  link = soup.find("a")
  print(link["href"])
  ```
  
  ## Loop through items
  ```python
  for p in soup.find_all("p"):
      print(p.text)
  ```
  
  ## Graph: parsing flow
  ```mermaid
  flowchart LR
    A[HTML Text] --> B[BeautifulSoup]
    B --> C[Search Tags]
    C --> D[Extract Data]
  ```
  
  ## Remember
  - BeautifulSoup builds a tree from HTML
  - Use find / find_all
  - .text gets text
  - ["attr"] gets attribute
  

Learn how to parse HTML and extract data using BeautifulSoup with clear searching patterns and examples.

HTML Parsing with BeautifulSoup

Now we combine everything into one real scraper.
  
  Goal: scrape all links from a page.
  
  ## Step 1: Request page
  ```python
  import requests
  from bs4 import BeautifulSoup
  
  url = "https://example.com"
  res = requests.get(url)
  
  if res.status_code != 200:
      print("Failed")
      exit()
  ```
  
  ## Step 2: Parse HTML
  ```python
  soup = BeautifulSoup(res.text, "html.parser")
  ```
  
  ## Step 3: Find links
  ```python
  links = soup.find_all("a")
  ```
  
  ## Step 4: Extract and clean
  ```python
  for a in links:
      text = a.text.strip()
      href = a.get("href")
      print(text, "->", href)
  ```
  
  ## Full script
  ```python
  import requests
  from bs4 import BeautifulSoup
  
  url = "https://example.com"
  res = requests.get(url)
  
  if res.status_code == 200:
      soup = BeautifulSoup(res.text, "html.parser")
      for a in soup.find_all("a"):
          print(a.text.strip(), "->", a.get("href"))
  else:
      print("Request failed")
  ```
  
  ## Graph: full scraper
  ```mermaid
  flowchart TD
    A[Start] --> B[Request URL]
    B --> C{Status 200?}
    C -->|Yes| D[Parse HTML]
    D --> E[Find tags]
    E --> F[Extract data]
    F --> G[Print/Save]
    C -->|No| H[Stop]
  ```
  
  ## What you learned
  - Sending request
  - Parsing HTML
  - Finding elements
  - Extracting data
  
  ## Remember
  This pattern repeats in every scraper:
  Request → Parse → Find → Extract → Save
  

Build your first real scraper step by step: request a page, parse HTML, extract multiple items, and print clean results.

Your First Complete Scraper

Scraping is useless if you don’t save data.
  
  Your goal is not just to print results, but to:
  - store them
  - reuse later
  - analyze
  - share
  
  ## Why saving matters
  If you scrape:
  - today: 100 products
  - tomorrow: prices change  
  
  You want history.
  
  ## Common formats
  - CSV: simple tables (Excel friendly)
  - JSON: structured/nested data (APIs, configs)
  
  ---
  
  ## Save to CSV
  CSV = comma separated values.
  
  ```python
  import csv
  
  data = [
    ["Name", "Price"],
    ["Apple", 120],
    ["Banana", 80],
  ]
  
  with open("products.csv", "w", newline="", encoding="utf-8") as f:
      writer = csv.writer(f)
      writer.writerows(data)
  ```
  
  You can open this file in Excel.
  
  ---
  
  ## Save scraped data to CSV
  ```python
  import requests, csv
  from bs4 import BeautifulSoup
  
  res = requests.get("https://example.com")
  soup = BeautifulSoup(res.text, "html.parser")
  
  rows = []
  rows.append(["Text", "Link"])
  
  for a in soup.find_all("a"):
      rows.append([a.text.strip(), a.get("href")])
  
  with open("links.csv", "w", newline="", encoding="utf-8") as f:
      writer = csv.writer(f)
      writer.writerows(rows)
  ```
  
  ---
  
  ## Save to JSON
  JSON is great for nested data.
  
  ```python
  import json
  
  data = [
    {"name": "Apple", "price": 120},
    {"name": "Banana", "price": 80}
  ]
  
  with open("products.json", "w", encoding="utf-8") as f:
      json.dump(data, f, indent=2)
  ```
  
  ---
  
  ## Graph: scraping to storage
  ```mermaid
  flowchart LR
    A[Scraper] --> B[Extracted Data]
    B --> C[CSV File]
    B --> D[JSON File]
  ```
  
  ## Remember
  - Always save data, not just print
  - CSV for tables
  - JSON for structured data
  - Use utf-8 encoding
  

Learn how to store scraped data properly in CSV and JSON files so it can be reused for analysis, reports, or databases.

Saving Scraped Data

Most websites show data across many pages.
  
  Example:
  - jobs page 1, 2, 3...
  - products page 1, 2, 3...
  
  You must loop through pages.
  
  ## Common pagination pattern
  URLs often look like:
  - site.com?page=1
  - site.com?page=2
  
  ## Simple loop example
  ```python
  import requests
  from bs4 import BeautifulSoup
  
  for page in range(1, 6):
      url = f"https://example.com?page={page}"
      res = requests.get(url)
      soup = BeautifulSoup(res.text, "html.parser")
  
      print("Page:", page)
      for h in soup.find_all("h2"):
          print(h.text.strip())
  ```
  
  ---
  
  ## Stop when no data found
  ```python
  page = 1
  while True:
      url = f"https://example.com?page={page}"
      res = requests.get(url)
      soup = BeautifulSoup(res.text, "html.parser")
  
      items = soup.find_all("h2")
      if not items:
          break
  
      for h in items:
          print(h.text.strip())
  
      page += 1
  ```
  
  ---
  
  ## Graph: pagination loop
  ```mermaid
  flowchart TD
    A[Start page=1] --> B[Request page]
    B --> C{Items found?}
    C -->|Yes| D[Extract data]
    D --> E[page += 1]
    E --> B
    C -->|No| F[Stop]
  ```
  
  ## Remember
  - Many sites use page numbers
  - Loop until no results
  - Add delays to be polite
  

Learn how to scrape data spread across many pages using pagination patterns and loops.

Scraping Multiple Pages

Real web scraping always faces errors:
  - network issues
  - server blocks
  - missing elements
  - broken HTML
  
  A good scraper must not crash.
  
  ## Handle request errors
  ```python
  import requests
  
  try:
      res = requests.get("https://example.com", timeout=10)
      res.raise_for_status()
  except requests.exceptions.RequestException as e:
      print("Request failed:", e)
  ```
  
  ---
  
  ## Safe element access
  ```python
  title = soup.find("h1")
  if title:
      print(title.text)
  else:
      print("Title not found")
  ```
  
  ---
  
  ## Retry logic
  ```python
  import time, requests
  
  def fetch(url, retries=3):
      for i in range(retries):
          try:
              res = requests.get(url, timeout=10)
              res.raise_for_status()
              return res
          except Exception as e:
              print("Retry", i+1, "failed")
              time.sleep(2)
      return None
  
  res = fetch("https://example.com")
  ```
  
  ---
  
  ## Graph: retry flow
  ```mermaid
  flowchart TD
    A[Request] --> B{Success?}
    B -->|Yes| C[Return data]
    B -->|No| D[Wait]
    D --> E[Retry]
    E --> B
  ```
  
  ## Remember
  - Never assume request will succeed
  - Always check elements
  - Retries make scraper robust
  

Make your scraper stable by handling network errors, timeouts, missing elements, and retrying failed requests.

Handling Errors and Retries

Scraping is powerful, but it must be responsible.
  
  ## What is robots.txt
  Most sites provide:
  site.com/robots.txt
  
  It tells bots:
  - which paths are allowed
  - which are disallowed
  
  Example:
  ```
  User-agent: *
  Disallow: /admin
  ```
  
  Means: do not scrape /admin.
  
  ## Why respect it
  - shows good behavior
  - avoids legal trouble
  - prevents IP bans
  
  ## Rate limiting
  Do not hit servers too fast.
  
  ```python
  import time
  
  for url in urls:
      requests.get(url)
      time.sleep(2)
  ```
  
  ## Identify yourself
  Use User-Agent:
  ```python
  headers = {
    "User-Agent": "MyScraper/1.0 (contact@example.com)"
  }
  ```
  
  ## Graph: ethical scraping
  ```mermaid
  flowchart TD
    A[Plan scrape] --> B[Check robots.txt]
    B --> C[Add delays]
    C --> D[Scrape responsibly]
  ```
  
  ## Remember
  - Always check robots.txt
  - Add delays
  - Do not overload servers
  - Scrape public data only
  

Learn responsible scraping: robots.txt, rate limiting, and how to avoid harming websites or getting blocked.

Ethics and robots.txt

Not all sites show data in HTML.
  
  Many modern sites load data using JavaScript.
  
  Your requests scraper gets:
  - empty HTML
  - no real data
  
  ## How to detect dynamic site
  If:
  - View Source shows no data
  - Inspect shows data loaded later
  
  Then it is dynamic.
  
  ## What really happens
  1) HTML loads empty
  2) JavaScript runs
  3) JS calls API
  4) Data appears in browser
  
  Your scraper only sees step 1.
  
  ## Graph: dynamic loading
  ```mermaid
  flowchart TD
    A[Request HTML] --> B[Empty Page]
    B --> C[JS runs in browser]
    C --> D[Fetch API data]
    D --> E[Render content]
  ```
  
  ## Solution options
  - Find the hidden API and scrape it
  - Or use browser automation:
    - Selenium
    - Playwright
  
  We will cover these later.
  
  ## Remember
  - requests + BeautifulSoup only works for static HTML
  - Many modern sites are dynamic
  - You must adapt your tools
  

Understand why some websites cannot be scraped with requests and BeautifulSoup, and when browser automation is needed.

Dynamic Websites Intro

Most websites allow searching or filtering:
  - product by category
  - jobs by city
  - news by keyword
  
  These filters are usually sent as **query parameters**.
  
  Example URL:
  site.com/search?q=python&page=2
  
  Here:
  q = python
  page = 2
  
  ## Why this matters
  Instead of clicking forms manually, you can:
  - change URL parameters
  - loop over values
  - collect large datasets
  
  ---
  
  ## Inspect query params
  Open browser DevTools → Network → search request.
  
  You may see:
  ?q=laptop&min_price=500
  
  ---
  
  ## Using params with requests
  ```python
  import requests
  
  url = "https://example.com/search"
  params = {
      "q": "python",
      "page": 1
  }
  
  res = requests.get(url, params=params)
  print(res.url)  # full URL with params
  ```
  
  ---
  
  ## Loop over search pages
  ```python
  for page in range(1, 4):
      params = {"q": "python", "page": page}
      res = requests.get("https://example.com/search", params=params)
      print("Scraping:", res.url)
  ```
  
  ---
  
  ## Graph: query param flow
  ```mermaid
  flowchart LR
    A[Search Form] --> B[URL Params]
    B --> C[requests.get()]
    C --> D[Filtered Results]
  ```
  
  ## Remember
  - Forms usually map to URL parameters
  - You can build URLs programmatically
  - This unlocks large datasets
  

Learn how websites pass search filters through URLs and forms, and how to scrape filtered results using query parameters.

Query Params and Search Forms

Websites remember you using **cookies**.
  
  Cookies store:
  - login info
  - preferences
  - session ids
  
  If you don’t keep them, the site may treat every request as new.
  
  ## Why sessions matter
  Without sessions:
  - you may get logged out
  - pagination may break
  - site may block you
  
  ---
  
  ## Using requests.Session
  ```python
  import requests
  
  session = requests.Session()
  session.headers.update({
    "User-Agent": "Mozilla/5.0"
  })
  
  res1 = session.get("https://example.com")
  res2 = session.get("https://example.com/profile")
  ```
  
  Both share cookies automatically.
  
  ---
  
  ## Inspect cookies
  ```python
  print(session.cookies)
  ```
  
  ---
  
  ## When to use sessions
  - after login
  - multi-step forms
  - sites with tracking
  
  ---
  
  ## Graph: session flow
  ```mermaid
  flowchart TD
    A[First Request] --> B[Receive Cookies]
    B --> C[Session Stores Cookies]
    C --> D[Next Request]
    D --> E[Same Identity]
  ```
  
  ## Remember
  - Session acts like a browser
  - Cookies maintain identity
  - Always use Session for multi-step scraping
  

Understand cookies and sessions, and learn how to maintain state across multiple requests like a real browser.

Sessions and Cookies

Some data is behind login:
  - dashboards
  - profiles
  - private listings
  
  To scrape it, you must:
  1) send login request
  2) keep session
  3) access protected pages
  
  ## Important note
  Only scrape accounts and data you own or have permission for.
  
  ---
  
  ## Step 1: Inspect login form
  In browser DevTools:
  - find form action URL
  - input names (username, password)
  
  Example:
  action: /login
  fields: email, password
  
  ---
  
  ## Step 2: Send POST with session
  ```python
  import requests
  
  session = requests.Session()
  
  login_url = "https://example.com/login"
  payload = {
      "email": "my@email.com",
      "password": "mypassword"
  }
  
  res = session.post(login_url, data=payload)
  
  print(res.status_code)
  ```
  
  ---
  
  ## Step 3: Access protected page
  ```python
  dashboard = session.get("https://example.com/dashboard")
  print(dashboard.text[:200])
  ```
  
  If you see dashboard HTML, login worked.
  
  ---
  
  ## Graph: login flow
  ```mermaid
  flowchart TD
    A[GET Login Page] --> B[Find form fields]
    B --> C[POST credentials]
    C --> D[Receive session cookie]
    D --> E[Access protected page]
  ```
  
  ## Remember
  - Use POST for login
  - Always use Session
  - Check response to confirm login
  

Learn the theory and practice of scraping pages behind login using POST requests and sessions.

Login Scraping Basics

Modern sites often load data from hidden APIs.
  
  Instead of scraping HTML:
  you can call the API directly.
  
  This is:
  - faster
  - cleaner
  - more stable
  
  ---
  
  ## How to find APIs
  Open DevTools → Network → XHR/Fetch.
  
  Look for:
  - JSON responses
  - endpoints like /api/items
  
  ---
  
  ## Example API request
  ```python
  import requests
  
  url = "https://example.com/api/products"
  res = requests.get(url)
  data = res.json()
  
  for p in data["items"]:
      print(p["name"], p["price"])
  ```
  
  ---
  
  ## With headers
  ```python
  headers = {
    "User-Agent": "Mozilla/5.0",
    "Accept": "application/json"
  }
  
  res = requests.get(url, headers=headers)
  ```
  
  ---
  
  ## Graph: API vs HTML
  ```mermaid
  flowchart LR
    A[Browser] --> B[API Call]
    B --> C[JSON Data]
    C --> D[Render HTML]
  
    E[Scraper] --> B
  ```
  
  ## Remember
  - Always check Network tab
  - API scraping is best when available
  - JSON is easier than HTML
  

Learn how many modern websites load data from APIs and how to scrape those APIs instead of parsing HTML.

Scraping APIs Directly

Before scraping, you must understand how web pages are built.

Web pages are written in **HTML**.  
Scraping means reading this structure and picking what you need.

## Why HTML knowledge is critical
If you don't understand:
- tags
- nesting
- attributes  
you will not know where your data lives.

## Basic HTML example
```html
<div class="product">
  <h2 class="title">Laptop</h2>
  <span class="price">$900</span>
</div>
```

Here:
- `div` wraps a product
- `h2` has the name
- `span` has the price

## Tree structure of HTML
HTML is a tree, not flat text.

```mermaid
flowchart TD
  A[div.product] --> B[h2.title]
  A --> C[span.price]
```

## How scraper sees this
You search:
- tag name
- class
- id
- path

Example with BeautifulSoup:
```python
product = soup.find("div", class_="product")
title = product.find("h2", class_="title").text
price = product.find("span", class_="price").text
```

## Key idea
You are not scraping a page.
You are navigating a tree.

## Remember
- Always inspect HTML first
- Identify container blocks
- Then target child elements


Learn how HTML is structured and how understanding tags, attributes, and nesting is the foundation of all web scraping.

HTML Structure for Scraping

CSS selectors let you describe elements exactly like front-end developers do.

They are powerful and concise.

## Common selectors
- `div` → all div tags
- `.price` → class="price"
- `#main` → id="main"
- `div span` → span inside div
- `div > span` → direct child

## Example HTML
```html
<div class="card">
  <h3 class="name">Phone</h3>
  <span class="price">$500</span>
</div>
```

## Using select() in BeautifulSoup
```python
card = soup.select_one("div.card")
name = card.select_one("h3.name").text
price = card.select_one("span.price").text
```

## Selector flow
```mermaid
flowchart LR
  A[HTML Page] --> B[CSS Selector]
  B --> C[Matched Elements]
```

## Why selectors matter
They:
- reduce code
- make scraping readable
- avoid deep nesting logic

## Remember
- Learn selectors once, use everywhere
- Prefer select/select_one for clarity


Master CSS selectors to target elements precisely when scraping with BeautifulSoup or Selenium.

CSS Selectors for Scraping

Most real websites split data into pages.

If you scrape only one page, you miss most data.

## Common pagination styles
1) ?page=1, ?page=2
2) /page/1, /page/2
3) Next button links
4) Infinite scroll (JS)

## URL based pagination example
```python
import requests
from bs4 import BeautifulSoup

for page in range(1, 6):
    url = f"https://example.com/products?page={page}"
    res = requests.get(url)
    soup = BeautifulSoup(res.text, "html.parser")

    for item in soup.select(".product"):
        print(item.select_one(".title").text)
```

## Pagination loop idea
```mermaid
flowchart TD
  A[Page 1] --> B[Scrape]
  B --> C[Next Page]
  C --> B
  C --> D[Stop when no data]
```

## Key checks
- Stop when no items found
- Or when next link missing

## Remember
- Always inspect pagination pattern
- Never hardcode too many pages blindly


Learn how to scrape data spread across multiple pages using page numbers, next links, or infinite scrolling patterns.

Pagination Handling

Real websites are messy.

Some items miss fields:
- no price
- no rating
- broken tags

Your scraper must not crash.

## Bad approach
```python
price = item.select_one(".price").text  # may crash
```

## Safe approach
```python
tag = item.select_one(".price")
price = tag.text.strip() if tag else "N/A"
```

## Wrap in try-except
```python
try:
    price = item.select_one(".price").text.strip()
except AttributeError:
    price = "N/A"
```

## Flow
```mermaid
flowchart TD
  A[Find Tag] --> B{Exists?}
  B -->|Yes| C[Read Text]
  B -->|No| D[Use Default]
```

## Remember
- Missing data is normal
- Always code defensively


Learn how to safely handle missing tags and inconsistent HTML so your scraper never crashes.

Handling Missing Data

Websites inspect headers to decide who you are.

If no User-Agent:
They think you are a bot.

## What is User-Agent
It tells:
- browser
- OS
- device

Example:
Mozilla/5.0 (Windows NT 10.0; Win64; x64)...

## Always send headers
```python
headers = {
  "User-Agent": "Mozilla/5.0"
}

res = requests.get("https://example.com", headers=headers)
```

## Request flow
```mermaid
flowchart LR
  A[Scraper] -->|Headers| B[Server]
  B --> C{Looks Real?}
  C -->|Yes| D[Send Page]
  C -->|No| E[Block]
```

## Remember
- User-Agent is mandatory
- Mimic a real browser


Understand HTTP headers, especially User-Agent, and why many sites block requests without them.

User-Agent and Headers

Sending many requests fast looks suspicious.

It can:
- overload servers
- get you blocked
- break your scraper

## Always add delays
```python
import time

for url in urls:
    requests.get(url)
    time.sleep(2)  # wait 2 seconds
```

## Why delay matters
Humans click slowly.
Bots don't.

## Delay flow
```mermaid
flowchart LR
  A[Request] --> B[Wait]
  B --> C[Next Request]
```

## Remember
- Respect server load
- Slow scrapers survive longer


Learn why sending requests too fast is dangerous and how to use delays to scrape responsibly.

Rate Limiting and Delays

Scraping is useless if you don’t save data.

CSV is simple and universal.

## Writing CSV
```python
import csv

rows = [
  ["Name", "Price"],
  ["Laptop", "900"],
  ["Phone", "500"]
]

with open("products.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerows(rows)
```

## Graph
```mermaid
flowchart LR
  A[Scraped Data] --> B[CSV File]
  B --> C[Excel / Analysis]
```

## Remember
- CSV works everywhere
- Always add headers


Learn how to store scraped data into CSV files for Excel and analysis.

Saving Data to CSV

JSON is best for structured and nested data.

## Write JSON
```python
import json

data = [
  {"name": "Laptop", "price": 900},
  {"name": "Phone", "price": 500}
]

with open("products.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)
```

## Flow
```mermaid
flowchart LR
  A[Python Dict/List] --> B[JSON File]
```

## Remember
- JSON keeps structure
- Ideal for APIs


Store structured scraped data into JSON format for APIs and modern applications.

Saving Data to JSON

Many sites use forms to filter data.

Behind forms are URL parameters.

## Example search URL
https://example.com/search?q=python&page=1

## Requests with params
```python
params = {
  "q": "python",
  "page": 1
}

res = requests.get("https://example.com/search", params=params)
```

## Graph
```mermaid
flowchart LR
  A[Params] --> B[Request URL]
  B --> C[Filtered Page]
```

## Remember
- Inspect form action URL
- Rebuild it with params


Learn how to scrape search results and form-based pages using query parameters.

Scraping Forms and Queries

Some sites need login.

Sessions keep cookies between requests.

## Using Session
```python
import requests

s = requests.Session()

s.post("https://example.com/login", data={
  "user": "abc",
  "pass": "123"
})

res = s.get("https://example.com/dashboard")
print(res.text)
```

## Flow
```mermaid
flowchart LR
  A[Login] --> B[Session]
  B --> C[Authenticated Requests]
```

## Remember
- Session keeps state
- Needed for logged pages


Maintain cookies and login sessions while scraping using requests.Session.

Session Handling

Scrapers break often because sites change.

You must know how to debug.

## Check status code
```python
res = requests.get(url)
print(res.status_code)
```

## Print partial HTML
```python
print(res.text[:500])
```

## Save page for inspection
```python
with open("page.html", "w", encoding="utf-8") as f:
    f.write(res.text)
```

Open it in browser and inspect.

## Debug flow
```mermaid
flowchart TD
  A[Scraper Fails] --> B[Check Status]
  B --> C[View HTML]
  C --> D[Update Selectors]
```

## Remember
- Websites change
- Debugging is part of scraping


Learn how to debug broken scrapers using status codes, printing HTML, and browser inspection.

Debugging Scrapers

When you scrape a site, you talk directly to its server.

Servers try to answer:
“Is this a human or a bot?”

This is why anti-bot systems exist.

## Why sites block bots
- protect server load
- stop data abuse
- prevent scraping of private data
- avoid spam and attacks

## Common bot signals
Websites look for:
- too many requests per second
- missing or fake headers
- same pattern again and again
- no JavaScript execution
- repeated hits from one IP

If detected, they may:
- return 403 or 429
- show CAPTCHA
- redirect to warning page

## What a good scraper should do
Not fight, but behave like a human:
- send real headers
- use delays
- follow robots.txt
- avoid restricted paths
- scrape only what you need

## Example: adding headers and delay
```python
import time
import requests

headers = {
  "User-Agent": "Mozilla/5.0"
}

res = requests.get("https://example.com", headers=headers)
time.sleep(3)
```

## Graph: anti-bot decision
```mermaid
flowchart LR
  A[Scraper Request] --> B[Website]
  B --> C{Human-like?}
  C -->|Yes| D[Send Page]
  C -->|No| E[Block / CAPTCHA]
```

## Remember
- Anti-bot is protection, not challenge
- Respect sites and scrape politely
- Human-like behavior keeps scrapers alive


Understand how websites detect bots, what signals they use, and how to design scrapers that behave like respectful human visitors.

Anti-Bot Systems

Every serious scraper must respect robots.txt.

It tells bots which parts of a site are allowed.

## What is robots.txt
It lives at:
https://site.com/robots.txt

Example:
User-agent: *
Disallow: /admin
Allow: /products

Means:
- all bots allowed on /products
- no bots on /admin

## Why you should follow it
- legal safety
- professional practice
- avoid being banned
- show respect

## Check robots.txt using Python
```python
import urllib.robotparser as robotparser

rp = robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

print(rp.can_fetch("*", "https://example.com/products"))
```

## Graph: robots decision
```mermaid
flowchart TD
  A[URL to scrape] --> B[Check robots.txt]
  B -->|Allowed| C[Scrape]
  B -->|Disallowed| D[Stop]
```

## Remember
- Always check robots.txt first
- Never scrape disallowed paths
- Ethics matter in scraping


Learn how to read and use robots.txt files to know what you are allowed to scrape and build ethical scrapers.

Robots.txt in Practice

A proxy is an intermediate server between you and a website.

Instead of:
You → Website  
It becomes:
You → Proxy → Website

## Why proxies are used
- distribute traffic
- avoid IP-based limits
- access region content
- improve reliability

## Important warning
Proxies are tools, not magic.
Do not use them to break rules.

## Example: requests with proxy
```python
import requests

proxies = {
  "http": "http://127.0.0.1:8080",
  "https": "http://127.0.0.1:8080"
}

res = requests.get("https://example.com", proxies=proxies, timeout=10)
print(res.status_code)
```

## Graph: proxy path
```mermaid
flowchart LR
  A[Scraper] --> B[Proxy Server]
  B --> C[Website]
  C --> B --> A
```

## When to use proxies
- scraping large scale
- site limits per IP
- geo-blocked pages

## Remember
- Use trusted proxies only
- Expect slower speed
- Avoid sensitive data


Understand what proxies are, when they are needed, and how to send scraping requests through proxies safely.

Proxy Usage

Some pages load data using JavaScript.

If you fetch them with requests:
You may see empty HTML.

## Why this happens
Browser loads page →
JavaScript runs →
JS fetches data →
Page updates

Requests only gets initial HTML.

## How to detect JS pages
- View page source: data missing
- Inspect in browser: data appears
- Network tab shows API calls

## Solution: use browser automation
Tools:
- Selenium
- Playwright

## Simple Selenium example
```python
from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com")

title = driver.find_element(By.CLASS_NAME, "title").text
print(title)

driver.quit()
```

## Graph: JS rendering
```mermaid
flowchart TD
  A[HTML] --> B[Browser]
  B --> C[Run JS]
  C --> D[Final DOM]
  D --> E[Scrape]
```

## Remember
- requests cannot run JS
- Use Selenium only when needed
- Always check if API exists first


Learn why some pages look empty to requests and how JavaScript rendering changes scraping using tools like Selenium or Playwright.

JavaScript Rendered Pages

As scrapers grow, one-file scripts become messy.

Good scrapers are designed like systems.

## Three main layers

1) Fetch → get pages
2) Parse → extract data
3) Store → save results

## Why separate layers
- easier debugging
- reusable code
- easy changes when site updates

## Example structure
```python
def fetch(url):
    return requests.get(url, headers=headers).text

def parse(html):
    soup = BeautifulSoup(html, "html.parser")
    return soup.select_one(".title").text

def store(data):
    print(data)  # or save to file/db
```

```python
html = fetch(url)
data = parse(html)
store(data)
```

## Graph: architecture
```mermaid
flowchart LR
  A[Fetch] --> B[Parse]
  B --> C[Store]
```

## Real projects add
- retries
- logging
- error handling
- queues

## Remember
- Design early, save pain later
- Clean layers make scrapers robust


Design a clean scraping system: separation of fetch, parse, and store layers for maintainable and scalable scrapers.

Scraping Architecture

Real networks are unreliable.

Requests may fail because of:
- slow servers
- connection drops
- temporary blocks
- timeouts

A good scraper must not crash on first failure.

## Common errors
- Timeout
- ConnectionError
- 5xx server errors
- 429 Too Many Requests

## Basic try-except
```python
import requests

try:
    res = requests.get("https://example.com", timeout=10)
    res.raise_for_status()
except requests.exceptions.RequestException as e:
    print("Request failed:", e)
```

## Retry logic (simple loop)
```python
import time
import requests

def fetch_with_retry(url, retries=3, delay=3):
    for attempt in range(1, retries+1):
        try:
            res = requests.get(url, timeout=10)
            res.raise_for_status()
            return res.text
        except Exception as e:
            print(f"Attempt {attempt} failed:", e)
            time.sleep(delay)
    return None

html = fetch_with_retry("https://example.com")
```

## Smarter retries with backoff
```python
import time
import requests

def fetch(url, retries=4):
    delay = 2
    for i in range(retries):
        try:
            return requests.get(url, timeout=10).text
        except:
            time.sleep(delay)
            delay *= 2
```

## Graph: retry flow
```mermaid
flowchart TD
  A[Request] --> B{Success?}
  B -->|Yes| C[Return]
  B -->|No| D[Wait]
  D --> A
```

## Remember
- Always set timeouts
- Retry only temporary errors
- Log failures for debugging


Learn how to handle network errors, timeouts, and temporary failures using retries so your scraper becomes stable and reliable.

Retries and Error Handling

Websites limit how fast you can hit them.

This is called rate limiting.

## Why rate limits exist
- protect servers
- ensure fair use
- stop abuse

If ignored, you may get:
- 429 errors
- IP bans
- blocked accounts

## Simple manual delay
```python
import time
import requests

urls = ["https://example.com/page1", "https://example.com/page2"]

for url in urls:
    res = requests.get(url)
    print(res.status_code)
    time.sleep(2)
```

## Token bucket style (concept)
You allow N requests per time window.

## Using a simple limiter function
```python
import time

last_call = 0
def rate_limit(seconds):
    global last_call
    now = time.time()
    wait = seconds - (now - last_call)
    if wait > 0:
        time.sleep(wait)
    last_call = time.time()
```

```python
for url in urls:
    rate_limit(2)
    requests.get(url)
```

## Graph: throttling
```mermaid
flowchart LR
  A[Request] --> B[Wait]
  B --> C[Next Request]
```

## Remember
- Always be slower than needed
- Respect site limits if documented
- Slow scraping is safer scraping


Understand rate limits, why sites enforce them, and how to throttle your scraper to stay within safe request speeds.

Rate Limiting

When scraping many pages, waiting one by one is slow.

Concurrency lets you fetch many pages at the same time.

## Two common models
1) Threads (requests + threading)
2) Async IO (aiohttp + asyncio)

## Why it helps
Network waits dominate scraping.
While one waits, others can run.

## Threaded example
```python
import requests
from concurrent.futures import ThreadPoolExecutor

urls = ["https://example.com"] * 10

def fetch(url):
    return requests.get(url).status_code

with ThreadPoolExecutor(max_workers=5) as pool:
    results = list(pool.map(fetch, urls))

print(results)
```

## Async example with aiohttp
```python
import asyncio
import aiohttp

async def fetch(session, url):
    async with session.get(url) as res:
        return res.status

async def main(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, u) for u in urls]
        return await asyncio.gather(*tasks)

urls = ["https://example.com"] * 10
print(asyncio.run(main(urls)))
```

## Graph: async flow
```mermaid
flowchart TD
  A[Task 1] --> C[Event Loop]
  B[Task 2] --> C
  C --> D[Responses]
```

## Important cautions
- Still respect rate limits
- Too many threads can overload
- Async code is harder to debug

## Remember
- Use concurrency for large scraping jobs
- Threading is easier to start
- Async scales better for huge workloads


Speed up scraping with concurrency: understand threads vs async, when to use them, and how to fetch many pages efficiently.

Concurrency and Async Scraping

When a scraper runs for hours, print() is not enough.

You need logs.

## Why logging matters
- know what failed
- know when it failed
- keep history
- debug later

## Basic logging setup
```python
import logging

logging.basicConfig(
    filename="scraper.log",
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s"
)

logging.info("Scraper started")
```

## Log during scraping
```python
try:
    res = requests.get("https://example.com")
    logging.info("Fetched page successfully")
except Exception as e:
    logging.error("Fetch failed: %s", e)
```

## Log progress
```python
for i, url in enumerate(urls, 1):
    logging.info("Processing %d/%d: %s", i, len(urls), url)
```

## Graph: logging pipeline
```mermaid
flowchart LR
  A[Scraper] --> B[Logger]
  B --> C[Log File]
  C --> D[Review & Debug]
```

## Remember
- Log start, success, and errors
- Use files for long jobs
- Logs are your black box recorder


Add proper logging to track scraper activity, errors, and progress so long-running jobs can be monitored and debugged.

Logging and Monitoring

Websites change often.

Your scraper may break without warning.

Testing helps catch this early.

## Key idea
Separate:
- fetch logic
- parse logic

Then test parsing with saved HTML.

## Example: parse function
```python
from bs4 import BeautifulSoup

def parse_title(html):
    soup = BeautifulSoup(html, "html.parser")
    return soup.select_one("h1").text.strip()
```

## Test with sample HTML
```python
def test_parse_title():
    html = "<html><h1>Hello</h1></html>"
    assert parse_title(html) == "Hello"

test_parse_title()
```

## Save real HTML for tests
```python
with open("sample.html") as f:
    html = f.read()

assert parse_title(html) == "Expected Title"
```

## Why this helps
- no network needed
- fast tests
- safe against site downtime

## Graph: test flow
```mermaid
flowchart LR
  A[HTML Sample] --> B[Parse Function]
  B --> C[Expected Data]
```

## Remember
- Always test parsing logic
- Keep sample HTML files
- Tests save hours of debugging


Learn how to test scraping logic using saved HTML and unit tests so site changes don’t silently break your scrapers.

Testing Scrapers

Scraping is useless if you don’t store the data properly.

Real goal:
- collect
- clean
- store
- analyze

## Common storage options
1) CSV → spreadsheets, simple data
2) JSON → APIs, nested data
3) Database → large or long-term data

---

## 1) Store to CSV

```python
import csv

rows = [
    ["Name", "Price"],
    ["Book A", "10"],
    ["Book B", "15"]
]

with open("products.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerows(rows)
```

Why CSV:
- easy to open in Excel
- simple format
- good for tables

---

## 2) Store to JSON

```python
import json

data = [
    {"name": "Book A", "price": 10},
    {"name": "Book B", "price": 15}
]

with open("products.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)
```

Why JSON:
- keeps structure
- good for nested data
- APIs use it

---

## 3) Store in SQLite database

```python
import sqlite3

conn = sqlite3.connect("data.db")
cur = conn.cursor()

cur.execute("""
CREATE TABLE IF NOT EXISTS products (
  name TEXT,
  price REAL
)
""")

cur.execute("INSERT INTO products VALUES (?, ?)", ("Book A", 10))
cur.execute("INSERT INTO products VALUES (?, ?)", ("Book B", 15))

conn.commit()
conn.close()
```

Why database:
- handles large data
- fast queries
- avoids duplicate rows

---

## Graph: data pipeline

```mermaid
flowchart LR
  A[Scraper] --> B[Parsed Data]
  B --> C[CSV]
  B --> D[JSON]
  B --> E[Database]
```

## Remember
- Always save data in structured form
- CSV for simple tables
- JSON for nested records
- DB for serious projects


Learn how to store scraped data properly in CSV, JSON, and databases so your scraping work becomes useful for analysis and applications.

Storing Scraped Data

Scraped data is almost never clean.

You will see:
- extra spaces
- missing values
- broken text
- wrong formats

Cleaning is mandatory.

---

## Example raw data

```python
raw = {
  "name": "  Book A \n",
  "price": " $10.00 ",
  "rating": "N/A"
}
```

---

## Clean text

```python
name = raw["name"].strip()
price = raw["price"].replace("$", "").strip()
rating = raw["rating"]

if rating == "N/A":
    rating = None
```

---

## Convert types safely

```python
def to_float(val):
    try:
        return float(val)
    except:
        return None

price = to_float(price)
```

---

## Validate fields

```python
if not name:
    raise ValueError("Name missing")

if price is None:
    print("Price invalid")
```

---

## Example clean record

```python
clean = {
  "name": name,
  "price": price,
  "rating": rating
}
print(clean)
```

---

## Graph: cleaning flow

```mermaid
flowchart LR
  A[Raw HTML Data] --> B[Extract]
  B --> C[Clean]
  C --> D[Validate]
  D --> E[Store]
```

## Remember
- Always strip and normalize text
- Convert strings to numbers
- Handle missing values
- Bad data breaks analysis later


Learn how to clean messy scraped data, remove noise, validate fields, and prepare it for analysis or storage.

Data Cleaning and Validation

Most modern sites try to detect bots.

If detected:
- 403 errors
- CAPTCHA
- IP ban
- blocked pages

Goal:
Behave like a normal user.

---

## Use real headers

```python
headers = {
  "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
  "Accept-Language": "en-US,en;q=0.9"
}

requests.get("https://example.com", headers=headers)
```

---

## Keep a session

```python
import requests

session = requests.Session()
session.headers.update(headers)

res = session.get("https://example.com")
```

Why:
- keeps cookies
- looks more human

---

## Add random delays

```python
import time, random

time.sleep(random.uniform(2, 5))
```

---

## Avoid patterns
Bad:
- same delay every time
- same URL order
- too many requests per second

---

## Graph: detection logic

```mermaid
flowchart TD
  A[Bot Requests] --> B{Looks human?}
  B -->|Yes| C[Allow]
  B -->|No| D[Block]
```

## Remember
- Always respect site rules
- Slow and steady wins
- Never try to bypass serious protections


Understand how websites detect scrapers and learn ethical ways to reduce blocking using headers, delays, and session behavior.

Avoiding Blocks and Detection

When scraping grows from 100 pages to 100,000 pages,
simple scripts fail.

You need structure.

---

## Problems at scale
- crashes waste hours
- duplicate scraping
- memory overflow
- no resume support

---

## Core ideas
1) Split work into tasks
2) Use workers
3) Save progress
4) Resume on failure

---

## Simple worker pattern

```python
from collections import deque

queue = deque(["url1", "url2", "url3"])

while queue:
    url = queue.popleft()
    print("Processing", url)
```

---

## Save checkpoints

```python
done = set()

def process(url):
    print("Scrape", url)

for url in urls:
    if url in done:
        continue
    process(url)
    done.add(url)
```

---

## Use files or DB for progress
Store processed URLs so you can restart safely.

---

## Graph: scalable system

```mermaid
flowchart LR
  A[URL List] --> B[Queue]
  B --> C[Worker 1]
  B --> D[Worker 2]
  B --> E[Worker 3]
  C --> F[Storage]
  D --> F
  E --> F
```

## Remember
- Always design for failure
- Save progress often
- Split work into small tasks


Learn how to design scraping systems for thousands of pages using workers, queues, and checkpoints so jobs can run reliably.

Scraping at Scale

Scraping power comes with responsibility.

You must know:
What is allowed.
What is risky.
What is illegal.

---

## Check robots.txt
Example:
https://example.com/robots.txt

If disallowed, do not scrape.

---

## Read Terms of Service
Many sites forbid scraping.
Respect them.

---

## Avoid personal data
Never collect:
- emails
- phone numbers
- passwords
- private profiles

---

## Ethical rules
- scrape slowly
- don’t overload servers
- identify yourself if possible
- stop if asked

---

## Real world use cases
Good:
- price comparison
- research
- public datasets

Bad:
- stealing content
- bypassing paywalls
- abusing systems

---

## Graph: responsible scraping

```mermaid
flowchart TD
  A[Want to Scrape] --> B{Allowed?}
  B -->|Yes| C[Scrape Responsibly]
  B -->|No| D[Do Not Scrape]
```

## Remember
- Legal issues can be serious
- Always prefer official APIs
- Be a responsible developer


Understand ethical and legal boundaries of web scraping: robots.txt, terms of service, personal data, and responsible usage.

Ethics and Legal Use

In real projects, scraped data is stored in databases.

Why databases:
- handle large data
- fast search and filters
- avoid duplicates
- multi-user access
- long-term storage

This lesson teaches:
- table design
- insert/update logic
- duplicate handling

---

## Example use case
Scrape products:
- name
- price
- url
- scraped_at

---

## Step 1: Design table

```sql
CREATE TABLE IF NOT EXISTS products (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  name TEXT,
  price REAL,
  url TEXT UNIQUE,
  scraped_at TEXT
);
```

Key idea:
- url is UNIQUE to avoid duplicates.

---

## Step 2: Connect to SQLite

```python
import sqlite3

conn = sqlite3.connect("scraped.db")
cur = conn.cursor()
```

---

## Step 3: Insert scraped item

```python
from datetime import datetime

def save_product(name, price, url):
    cur.execute("""
        INSERT OR IGNORE INTO products (name, price, url, scraped_at)
        VALUES (?, ?, ?, ?)
    """, (name, price, url, datetime.utcnow().isoformat()))
    conn.commit()
```

---

## Step 4: Use in scraper

```python
product = {"name": "Book A", "price": 10, "url": "http://x.com/a"}
save_product(product["name"], product["price"], product["url"])
```

---

## Graph: scraper to database

```mermaid
flowchart LR
  A[Scraper] --> B[Parsed Data]
  B --> C[SQL Insert]
  C --> D[Database]
```

---

## PostgreSQL note
For large systems use psycopg2 / asyncpg.
Logic stays same: connect → insert → commit.

---

## Remember
- Always design schema first
- Use UNIQUE keys to avoid duplicates
- Insert as you scrape, not after everything


Learn to design tables and store scraped data directly into a database using SQLite and PostgreSQL so scraping becomes production-ready.

Store Scraped Data in Database

Scraping is often repeated daily or hourly.

So you must:
- avoid inserting same data again
- update prices or fields if changed

This is called upsert (update or insert).

---

## Problem
Same product appears again with new price.

---

## SQLite upsert example

```python
def upsert_product(name, price, url):
    cur.execute("""
        INSERT INTO products (name, price, url, scraped_at)
        VALUES (?, ?, ?, datetime('now'))
        ON CONFLICT(url) DO UPDATE SET
          name = excluded.name,
          price = excluded.price,
          scraped_at = excluded.scraped_at
    """, (name, price, url))
    conn.commit()
```

---

## Use in loop

```python
for p in products:
    upsert_product(p["name"], p["price"], p["url"])
```

---

## Why this matters
- keeps data fresh
- no duplicates
- safe repeated runs

---

## Graph: upsert logic

```mermaid
flowchart TD
  A[New Item] --> B{URL exists?}
  B -->|No| C[Insert]
  B -->|Yes| D[Update]
```

---

## Remember
- Always identify a UNIQUE field
- Use upsert instead of blind insert
- Makes scraper idempotent (safe to rerun)


Learn how to update existing rows, avoid duplicates, and keep your scraped database fresh using upserts.

Handling Updates and Duplicates

Real scrapers run automatically:
- every hour
- every day
- every week

Manual runs are not practical.

---

## Option 1: cron (Linux)

Run every day at 2 AM:

```bash
0 2 * * * /usr/bin/python3 /path/scraper.py >> scraper.log 2>&1
```

Meaning:
minute hour day month weekday

---

## Option 2: Python schedule library

```python
import schedule
import time

def job():
    print("Running scraper...")

schedule.every().day.at("02:00").do(job)

while True:
    schedule.run_pending()
    time.sleep(60)
```

---

## Why scheduling matters
- keeps data fresh
- no human needed
- works 24/7

---

## Graph: scheduler flow

```mermaid
flowchart LR
  A[Clock] --> B[Scheduler]
  B --> C[Run Scraper]
  C --> D[Store in DB]
```

---

## Remember
- cron is reliable for servers
- Python scheduler good for apps
- Always log scheduled runs


Learn how to run scrapers automatically using cron and Python schedulers so data stays up-to-date without manual runs.

Scheduling Scraping Jobs

Never hardcode:
- database passwords
- API keys
- URLs

They change and must be protected.

---

## Use environment variables

```bash
export DB_URL="sqlite:///scraped.db"
export API_KEY="secret123"
```

```python
import os

db_url = os.getenv("DB_URL")
api_key = os.getenv("API_KEY")
```

---

## Use .env file (with python-dotenv)

```
DB_URL=sqlite:///scraped.db
API_KEY=secret123
```

```python
from dotenv import load_dotenv
load_dotenv()
```

---

## Why config matters
- safe secrets
- different envs (dev, prod)
- easy changes without code edit

---

## Graph: config flow

```mermaid
flowchart LR
  A[Env / Config File] --> B[App]
  B --> C[DB / APIs]
```

---

## Remember
- Never commit secrets to git
- Use env vars for sensitive data
- Keep config separate from logic


Learn how to manage settings and protect secrets like database URLs and API keys using config files and environment variables.

Configuration and Secrets

This lesson ties everything together.

Goal:
One system that:
- fetches pages
- parses data
- cleans fields
- stores in DB
- logs progress
- runs on schedule

---

## High-level pipeline

```mermaid
flowchart LR
  A[Scheduler] --> B[Fetcher]
  B --> C[Parser]
  C --> D[Cleaner]
  D --> E[DB Store]
  B --> F[Logger]
  C --> F
  D --> F
  E --> F
```

---

## Minimal pipeline code

```python
def run():
    html = fetch_with_retry(URL)
    items = parse(html)

    for item in items:
        clean = clean_item(item)
        upsert_product(
            clean["name"],
            clean["price"],
            clean["url"]
        )

if __name__ == "__main__":
    run()
```

---

## Why this matters
This structure:
- survives crashes
- avoids duplicates
- easy to maintain
- ready for scale

---

## Real-world usage
- price monitoring
- job listings
- news aggregation
- market research
- analytics feeds

---

## Remember
- Always think in pipeline
- Each step has one job
- This is how professional scrapers are built


Build a complete production scraping pipeline from request to database with retries, cleaning, logging, and scheduling.

End-to-End Scraping Pipeline

When scraping becomes serious, one-file scripts are not enough.

A clean structure helps you:
- debug faster
- reuse code
- add new scrapers easily
- work in teams

---

## Recommended folder layout

```
scraper/
│
├── main.py
├── fetcher.py
├── parser.py
├── storage.py
├── config.py
├── utils.py
├── requirements.txt
└── logs/
```

Meaning:
- fetcher: HTTP requests
- parser: HTML → data
- storage: DB logic
- config: URLs, settings
- utils: helpers

---

## Example: separation of concerns

### fetcher.py
```python
import requests

def fetch(url):
    return requests.get(url, timeout=10).text
```

### parser.py
```python
from bs4 import BeautifulSoup

def parse(html):
    soup = BeautifulSoup(html, "html.parser")
    return [h.text for h in soup.select("h2.title")]
```

### main.py
```python
from fetcher import fetch
from parser import parse

html = fetch("https://example.com")
items = parse(html)
print(items)
```

---

## Graph: clean flow

```mermaid
flowchart LR
  A[main.py] --> B[fetcher]
  B --> C[parser]
  C --> D[storage]
```

---

## Remember
- One file, one responsibility
- Clean structure saves time later
- This is how production scrapers are organized


Learn how to organize a scraping project with clear folders, reusable modules, and clean separation so your code stays maintainable as it grows.

Project Structure and Clean Code

In real systems, prints are not enough.

You need logs to know:
- when scraper ran
- what failed
- how many items scraped

---

## Why logging matters
Without logs:
- silent failures go unnoticed
- debugging becomes hard
- you lose trust in data

---

## Basic logging setup

```python
import logging

logging.basicConfig(
    filename="logs/scraper.log",
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s"
)

logging.info("Scraper started")
logging.error("Failed to fetch page")
```

---

## Use logs in pipeline

```python
logging.info("Fetching URL")
html = fetch(url)

logging.info("Parsing page")
items = parse(html)

logging.info(f"Found {len(items)} items")
```

---

## Example log output

```
2025-12-21 10:00:01 [INFO] Scraper started
2025-12-21 10:00:03 [INFO] Found 120 items
2025-12-21 10:00:04 [INFO] Data saved to DB
```

---

## Graph: logging flow

```mermaid
flowchart LR
  A[Scraper Step] --> B[Logger]
  B --> C[Log File]
```

---

## Remember
- Log start, end, and errors
- Logs are your eyes in production
- Always keep logs for scheduled jobs


Learn to log every scraping run, errors, and statistics so you always know what your scraper is doing in production.

Scrapers usually run on servers, not laptops.

Why deploy:
- always online
- scheduled runs
- stable IP
- long jobs

---

## Typical steps
1) Create server (AWS, Azure, DigitalOcean)
2) Install Python
3) Upload code
4) Install requirements
5) Setup cron

---

## Install dependencies

```bash
sudo apt update
sudo apt install python3 python3-pip -y
pip3 install -r requirements.txt
```

---

## Run manually

```bash
python3 main.py
```

---

## Add cron job

```bash
crontab -e
```

```
0 */6 * * * /usr/bin/python3 /home/user/scraper/main.py >> /home/user/scraper/logs/run.log 2>&1
```

Runs every 6 hours.

---

## Graph: deployment flow

```mermaid
flowchart LR
  A[Local Code] --> B[Server]
  B --> C[Cron]
  C --> D[Scraper]
  D --> E[Database]
```

---

## Remember
- Servers make scraping reliable
- Always redirect output to logs
- Test before scheduling


Learn how to run your scraper on a Linux server or cloud VM so it works 24/7 with scheduling and logs.

Deploy Scraper on Server

As targets grow, slow scrapers become a problem.

Goals:
- faster scraping
- no bans
- stable memory usage

---

## Key ideas
- reuse connections
- limit concurrency
- batch DB writes
- respect delays

---

## Example: concurrent requests with threads

```python
from concurrent.futures import ThreadPoolExecutor
import requests

def fetch(url):
    return requests.get(url).text

urls = ["https://example.com/1", "https://example.com/2"]

with ThreadPoolExecutor(max_workers=5) as ex:
    pages = list(ex.map(fetch, urls))

print(len(pages))
```

---

## Batch DB inserts

Instead of committing every row:
- collect 50–100 items
- insert in one transaction

This is much faster.

---

## Graph: scaling idea

```mermaid
flowchart LR
  A[URLs] --> B[Thread Pool]
  B --> C[Parsers]
  C --> D[Batch Insert]
  D --> E[Database]
```

---

## Important warning
More speed ≠ better scraping.

Always:
- keep delays
- rotate headers
- respect robots.txt

---

## Remember
- Scale carefully
- Measure before optimizing
- Stability is more important than raw speed


Learn how to speed up scraping safely using concurrency, batching, and efficient storage without getting blocked.

Performance and Scaling

Web scraping started simple:
- fetch HTML
- parse tags

Today:
- heavy JavaScript sites
- APIs behind logins
- bot detection
- legal rules

Tomorrow:
- more APIs
- AI-based extraction
- stronger protections

---

## Trends you must know

1) APIs over scraping  
Many sites now offer official APIs.

2) Dynamic sites  
More content loads via JS → tools like Playwright.

3) AI extraction  
LLMs can extract fields from messy HTML.

4) Legal awareness  
Data usage rules are becoming stricter.

---

## Modern pipeline idea

```mermaid
flowchart LR
  A[Fetcher / Browser] --> B[HTML]
  B --> C[AI / Parser]
  C --> D[Cleaner]
  D --> E[Database]
```

---

## Skills that keep you relevant
- HTML + DOM understanding
- API usage
- async Python
- data cleaning
- database design

---

## Career view
Scraping powers:
- analytics
- market research
- price intelligence
- lead generation
- monitoring systems

Demand remains strong where data matters.

---

## Final takeaway
Scraping is no longer just scripts.
It is:
- data engineering
- automation
- system design

If you master the full pipeline, you stay valuable.

---

## Remember
- Prefer APIs when available
- Keep learning modern tools
- Build scrapers as systems, not hacks


Understand where web scraping is heading: APIs, dynamic sites, AI extraction, legal boundaries, and how to stay relevant in coming years.

Future of Web Scraping

When scrapers move to production, security becomes critical.

You may deal with:
- API keys
- login credentials
- cookies
- private datasets

If leaked, they can cause real damage.

---

## Never hard-code secrets

Bad:
```python
API_KEY = "my-secret-key"
```

Good: use environment variables
```python
import os
API_KEY = os.getenv("API_KEY")
```

Run with:
```bash
export API_KEY="my-secret-key"
python main.py
```

---

## Use .env file (local dev)

```
API_KEY=my-secret-key
DB_PASS=strongpassword
```

```python
from dotenv import load_dotenv
load_dotenv()
```

---

## Validate external input
Never trust scraped or user data blindly.

```python
def clean(text):
    return text.strip()[:200]
```

---

## Avoid logging secrets
Do NOT log:
- passwords
- tokens
- cookies

---

## Graph: secure flow

```mermaid
flowchart LR
  A[Secrets] --> B[Env Variables]
  B --> C[Scraper]
  C --> D[Safe Usage]
```

---

## Remember
- Secrets go in env vars
- Sanitize all data
- Think like an attacker


Learn how to keep your scraper secure: protect credentials, avoid injections, handle secrets safely, and prevent accidental data leaks.

Security and Safe Scraping

Websites change often. Your scraper must not break silently.

Testing helps you:
- detect changes early
- avoid bad data
- refactor safely

---

## Idea: test parser, not network

Save sample HTML:
```python
sample_html = "<html><h2 class='title'>Item A</h2></html>"
```

Test parse logic:
```python
from parser import parse

def test_parse_titles():
    items = parse(sample_html)
    assert items == ["Item A"]
```

---

## Using pytest

```bash
pip install pytest
pytest
```

---

## Mock network calls

```python
from unittest.mock import patch

@patch("fetcher.fetch")
def test_pipeline(mock_fetch):
    mock_fetch.return_value = sample_html
    items = run()
    assert len(items) == 1
```

---

## Graph: test strategy

```mermaid
flowchart LR
  A[HTML Samples] --> B[Parser Tests]
  C[Mock Fetch] --> D[Pipeline Test]
```

---

## Remember
- Test parsing logic
- Mock network calls
- Catch breakages early


Learn how to test scraping logic so website changes do not silently break your pipeline, using sample HTML and unit tests.

A scraper without docs becomes useless after some months.

Good documentation answers:
- what it does
- how to run it
- how to configure
- where data goes
- common errors

---

## README example outline

```
# Project Name

## What it does
Scrapes product prices from example.com

## Setup
pip install -r requirements.txt

## Run
python main.py

## Config
Edit config.py

## Output
Data saved to SQLite DB

## Notes
Respect robots.txt
```

---

## Inline code comments

```python
# Fetch HTML from target URL
html = fetch(url)
```

---

## Graph: knowledge transfer

```mermaid
flowchart LR
  A[Developer] --> B[Docs]
  B --> C[Next Maintainer]
```

---

## Remember
- Write README
- Explain configs and outputs
- Your future self will thank you


Learn how to document your scraper so anyone can run, maintain, and extend it without confusion.

Documentation and Handover

Real scrapers live long. They need care.

Challenges:
- HTML changes
- network failures
- new blocks
- growing data

---

## Detect site change

```python
if not items:
    logging.warning("No items found, possible site change")
```

---

## Add alerts (simple)

```python
if errors > 5:
    send_email("Scraper failing!")
```

---

## Backup data
Always backup DB or files before overwriting.

---

## Graceful retry

```python
import time

for i in range(3):
    try:
        html = fetch(url)
        break
    except Exception:
        time.sleep(5)
```

---

## Graph: long-term cycle

```mermaid
flowchart LR
  A[Run] --> B[Store]
  B --> C[Monitor]
  C --> D[Fix]
  D --> A
```

---

## Remember
- Expect failures
- Monitor results
- Maintenance is part of scraping


Understand how to keep scrapers running for months or years: handling site changes, alerts, backups, and graceful recovery.

Maintenance and Long-Term Running

This is the final lesson of the Python Web Scraping course.

You did not just learn tools.
You learned how to build **data systems**.

---

## What you have mastered

### Foundations
- HTTP requests
- HTML and DOM
- BeautifulSoup
- XPath and selectors

### Dynamic scraping
- Selenium / Playwright
- waiting for JS content
- handling forms and clicks

### Robust pipelines
- retries and timeouts
- headers and sessions
- pagination
- login flows

### Data handling
- cleaning text
- validation
- saving to CSV, JSON
- storing in SQLite / PostgreSQL

### Production skills
- project structure
- logging
- deployment
- cron scheduling
- scaling
- security
- testing
- documentation
- maintenance

---

## Full pipeline you can now build

```mermaid
flowchart LR
  A[Target Site] --> B[Fetcher]
  B --> C[Parser]
  C --> D[Cleaner]
  D --> E[Database]
  E --> F[Reports / APIs]
```

---

## How to practice next
Build projects like:
- price tracker
- job scraper
- news aggregator
- lead collector
- monitoring dashboard

---

## Important rules to remember
- Prefer APIs when available
- Respect robots.txt and laws
- Do not overload servers
- Protect credentials
- Validate your data

---

## Career value
With these skills you can work in:
- data engineering
- automation
- analytics
- research
- backend systems

Scraping is a gateway skill to data pipelines.

---

## Final words
If you can:
- collect data
- clean it
- store it
- automate it
- maintain it

You can build real-world systems.

---

## Congratulations
You have completed the full **Python Web Scraping Course**.

Keep building.
Keep learning.
And turn data into value.


Complete the course with a full recap of all important topics, skills you gained, and how to apply them in real projects.