Scrapy Course – Python Web Scraping for Beginners

The Scrapy Beginners Course will teach you everything you need to learn to start scraping websites at scale using Python Scrapy. The course covers: - Creating your first Scrapy spider - Crawling through websites & scraping data from each page - Cleaning data with Items & Item Pipelines - Saving data to CSV files, MySQL & Postgres databases - Using fake user-agents & headers to avoid getting blocked - Using proxies to scale up your web scraping without getting banned - Deploying your scraper to the cloud & scheduling it to run periodically ✏️ Course created by Joe Kearney. ⭐️ Resources ⭐️ Course Resources - Scrapy Docs: https://docs.scrapy.org/en/latest/ - Course Guide: https://thepythonscrapyplaybook.com/freecodecamp-beginner-course/ - Course Github: https://github.com/orgs/python-scrapy-playbook/repositories - The Python Scrapy Playbook: https://thepythonscrapyplaybook.com/ Cloud Environments - Scrapyd: https://github.com/scrapy/scrapyd - ScrapydWeb: https://github.com/my8100/scrapydweb - ScrapeOps Monitor & Scheduler: https://scrapeops.io/monitoring-scheduling/ - Scrapy Cloud: https://www.zyte.com/scrapy-cloud/ Proxies - Proxy Plan Comparison Tool: https://scrapeops.io/proxy-providers/comparison/free-proxy-providers - ScrapeOps Proxy Aggregator: https://scrapeops.io/proxy-api-aggregator/ - Smartproxy: https://smartproxy.com/deals/proxyservers/ips ❤️ Try interactive Python courses we love, right in your browser: https://scrimba.com/freeCodeCamp-Python (Made possible by a grant from our friends at Scrimba) ⭐️ Contents ⭐️ ⌨️ (0:00:00) Part 1 - Scrapy & Course Introduction ⌨️ (0:08:22) Part 2 - Setup Virtual Env & Scrapy ⌨️ (0:16:28) Part 3 - Creating a Scrapy Project ⌨️ (0:28:17) Part 4 - Build your First Scrapy Spider ⌨️ (0:55:09) Part 5 - Build Discovery & Extraction Spider ⌨️ (1:20:11) Part 6 - Cleaning Data with Item Pipelines ⌨️ (1:44:19) Part 7 - Saving Data to Files & Databases ⌨️ (2:04:33) Part 8 - Fake User-Agents & Browser Headers ⌨️ (2:40:12) Part 9 - Rotating Proxies & Proxy APIs ⌨️ (3:18:12) Part 10 - Run Spiders in Cloud with Scrapyd ⌨️ (4:03:46) Part 11 - Run Spiders in Cloud with ScrapeOps ⌨️ (4:20:04) Part 12 - Run Spiders in Cloud with Scrapy Cloud ⌨️ (4:30:36) Part 13 - Conclusion & Next Steps 🎉 Thanks to our Champion and Sponsor supporters: 👾 davthecoder 👾 jedi-or-sith 👾 南宮千影 👾 Agustín Kussrow 👾 Nattira Maneerat 👾 Heather Wcislo 👾 Serhiy Kalinets 👾 Justin Hual 👾 Otis Morgan -- Learn to code for free and get a developer job: https://www.freecodecamp.org Read hundreds of articles on programming: https://freecodecamp.org/news