JavaScript Rendered Pages

Some pages load data using JavaScript.

If you fetch them with requests:
You may see empty HTML.

Why this happens

Browser loads page →
JavaScript runs →
JS fetches data →
Page updates

Requests only gets initial HTML.

How to detect JS pages

• View page source: data missing
• Inspect in browser: data appears
• Network tab shows API calls

Solution: use browser automation

Tools:

• Selenium
• Playwright

Simple Selenium example

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com")

title = driver.find_element(By.CLASS_NAME, "title").text
print(title)

driver.quit()

Graph: JS rendering

flowchart TD
  A[HTML] --> B[Browser]
  B --> C[Run JS]
  C --> D[Final DOM]
  D --> E[Scrape]

Remember

• requests cannot run JS
• Use Selenium only when needed
• Always check if API exists first