Dynamic Websites Intro

Not all sites show data in HTML.

Many modern sites load data using JavaScript.

Your requests scraper gets:

• empty HTML
• no real data

How to detect dynamic site

If:

• View Source shows no data
• Inspect shows data loaded later

Then it is dynamic.

What really happens

1. HTML loads empty
2. JavaScript runs
3. JS calls API
4. Data appears in browser

Your scraper only sees step 1.

Graph: dynamic loading

flowchart TD
  A[Request HTML] --> B[Empty Page]
  B --> C[JS runs in browser]
  C --> D[Fetch API data]
  D --> E[Render content]

Solution options

• Find the hidden API and scrape it
• Or use browser automation:
  • Selenium
  • Playwright

We will cover these later.

Remember

• requests + BeautifulSoup only works for static HTML
• Many modern sites are dynamic
• You must adapt your tools