Your First Complete Scraper

Now we combine everything into one real scraper.

Goal: scrape all links from a page.

Step 1: Request page

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
res = requests.get(url)

if res.status_code != 200:
    print("Failed")
    exit()

Step 2: Parse HTML

soup = BeautifulSoup(res.text, "html.parser")

Step 3: Find links

links = soup.find_all("a")

Step 4: Extract and clean

for a in links:
    text = a.text.strip()
    href = a.get("href")
    print(text, "->", href)

Full script

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
res = requests.get(url)

if res.status_code == 200:
    soup = BeautifulSoup(res.text, "html.parser")
    for a in soup.find_all("a"):
        print(a.text.strip(), "->", a.get("href"))
else:
    print("Request failed")

Graph: full scraper

flowchart TD
  A[Start] --> B[Request URL]
  B --> C{Status 200?}
  C -->|Yes| D[Parse HTML]
  D --> E[Find tags]
  E --> F[Extract data]
  F --> G[Print/Save]
  C -->|No| H[Stop]

What you learned

• Sending request
• Parsing HTML
• Finding elements
• Extracting data

Remember

This pattern repeats in every scraper:
Request → Parse → Find → Extract → Save