Pagination Handling

Most real websites split data into pages.

If you scrape only one page, you miss most data.

Common pagination styles

1. ?page=1, ?page=2
2. /page/1, /page/2
3. Next button links
4. Infinite scroll (JS)

URL based pagination example

import requests
from bs4 import BeautifulSoup

for page in range(1, 6):
    url = f"https://example.com/products?page={page}"
    res = requests.get(url)
    soup = BeautifulSoup(res.text, "html.parser")

    for item in soup.select(".product"):
        print(item.select_one(".title").text)

Pagination loop idea

flowchart TD
  A[Page 1] --> B[Scrape]
  B --> C[Next Page]
  C --> B
  C --> D[Stop when no data]

Key checks

• Stop when no items found
• Or when next link missing

Remember

• Always inspect pagination pattern
• Never hardcode too many pages blindly