Deploy Scraper on Server

Scrapers usually run on servers, not laptops.

Why deploy:

• always online
• scheduled runs
• stable IP
• long jobs

---

Typical steps

1. Create server (AWS, Azure, DigitalOcean)
2. Install Python
3. Upload code
4. Install requirements
5. Setup cron

---

Install dependencies

sudo apt update
sudo apt install python3 python3-pip -y
pip3 install -r requirements.txt

---

Run manually

python3 main.py

---

Add cron job

crontab -e

0 */6 * * * /usr/bin/python3 /home/user/scraper/main.py >> /home/user/scraper/logs/run.log 2>&1

Runs every 6 hours.

---

Graph: deployment flow

flowchart LR
  A[Local Code] --> B[Server]
  B --> C[Cron]
  C --> D[Scraper]
  D --> E[Database]

---

Remember

• Servers make scraping reliable
• Always redirect output to logs
• Test before scheduling