News Web Scraper
Parse the structure, not the text — and be a client the site would not mind.
- Python
- BeautifulSoup
- requests
About this project
A scraper that extracts headlines, links, sections and summaries from a news listing page and writes them to CSV. It ships with a sample page and runs entirely offline by default, so it works on any connection and without asking anything of a site you do not own.
The idea it is built around is that you parse the structure, not the text. A regular expression over HTML looks like it works and breaks the first time an attribute order changes or a tag gains a class — the README shows a specific input where the regex quietly returns the wrong thing and the parser keeps working.
The second theme is not optional and gets equal space: being a client the site would not mind. Identify yourself with a real User-Agent, leave a gap between requests, read robots.txt before fetching, and cache locally so re-running does not re-fetch. The project says plainly that scraping a site you do not own may breach its terms, and the bundled sample exists so that nobody has to in order to learn this.
What you will learn
- Parse HTML by structure with BeautifulSoup rather than pattern-matching it with a regex
- See a concrete case where the regex version breaks and the parser does not
- Write to CSV correctly, including fields that themselves contain commas
- Behave as a polite client: identify yourself, rate-limit, honour robots.txt and cache
- Resolve relative links against a base URL
Before you start
- Python basics — functions, lists and dictionaries
- Python 3.10 or newer, plus requests and beautifulsoup4 (one pip command)
- A little HTML — what a tag, a class and an attribute are
New to Python? The free Python course covers everything this project assumes.
Academic integrity notice
These projects are provided for learning, experimentation and reference. Understand the code, modify it for your own requirements, and acknowledge external work according to your institution’s academic policies. Priodemy does not supply work to be submitted as your own.
Features
- Runs offline against a bundled sample page — the default path makes no network request at all
- Structure-based parsing with BeautifulSoup, with the regex failure case documented
- CSV output with correct quoting for fields containing commas
- Politeness built in: User-Agent, rate limiting, robots.txt check and a local cache
- Relative links resolved against the page's declared base URL
- Pagination detected and reported rather than silently followed
Technology stack
Track: Python & AI/ML · Level: Starter
What is included
- scrape.py, parse.py and sample_page.html — commented throughout
- A README on parsing versus pattern-matching, and on scraping responsibly
- Suggested extensions, including following pagination and scraping a second site
Delivered as a 31 KB ZIP. SHA-256 published under Changelog so you can verify it.
System requirements
- Operating system
- Any
- Runtime
- Python 3.10 or newer. pip install requests beautifulsoup4 — only needed for the online mode.
Licence
Learn from it, change it, and use it in your own coursework and personal projects. Do not resell it or pass it off as your own original work.
- Personal and educational use
- Modifying the code however you like
- Using it as the basis of your own coursework, with honest acknowledgement
- Putting your modified version in a portfolio or on your own GitHub
- Reselling or redistributing the project, modified or not
- Presenting it as your own original work
- Commercial deployment — see the Commercial licence
Support window: 6 months from download.
Frequently asked
Do I need an internet connection?
No. The default run parses the bundled sample page and writes the CSV without touching the network — we tested that with all socket access blocked. Fetching a live URL is opt-in.
Is scraping legal?
It depends on the site and on where you are, and this project does not pretend otherwise. Many sites forbid it in their terms regardless of what robots.txt says. The README covers robots.txt, rate limiting and identifying yourself, and the sample page is there so you can learn the technique without pointing it at anyone.
Why not use a regular expression? It looks simpler.
It is simpler and it is wrong. HTML is not a regular language, so a pattern that matches today breaks when an attribute moves or a class is added. The README gives an input where exactly that happens, and you can run both versions against it.
Changelog
- First release
SHA-256cf0bccfab3529000574b18f53ca1327852fb187c12919c2caa4eccd7125a4313
You might also like
Free ATM Simulation System
A working cash machine in Python and MySQL — balance, withdrawal, deposit and a transaction history.
- Python
- MySQL
- mysql-connector
Free Student Management System
Add, search, update and report on student records with Python and MySQL.
- Python
- MySQL
- mysql-connector
Free Library Management System
Issue and return books, track members, and calculate fines — Python and MySQL.
- Python
- MySQL
- mysql-connector
Free Expense Manager (Python CLI)
The totals belong in the database, not in a Python loop — and the project times both to prove it.
- Python
- SQLite
- SQL