Pagination Handling

Most real websites split data into pages.

If you scrape only one page, you miss most data.

Common pagination styles 1) ?page=1, ?page=2 2) /page/1, /page/2 3) Next button links 4) Infinite scroll (JS)

URL based pagination example ```python import requests from bs4 import BeautifulSoup

for page in range(1, 6): url = f"https://example.com/products?page={page}" res = requests.get(url) soup = BeautifulSoup(res.text, "html.parser")

for item in soup.select(".product"): print(item.select_one(".title").text) ```

Pagination loop idea ```mermaid flowchart TD A[Page 1] --> B[Scrape] B --> C[Next Page] C --> B C --> D[Stop when no data] ```

Key checks - Stop when no items found - Or when next link missing

Remember - Always inspect pagination pattern - Never hardcode too many pages blindly