Web Scraping in Python: Getting Data Out of a Page Responsibly

Quick Answer

requests fetches the HTML and BeautifulSoup parses it. Check robots.txt and the terms of service first, prefer an official API if one exists, rate-limit yourself, and expect your selectors to break when the site changes.

Before you write any code

Three checks, in order, and they take two minutes.

Is there an API? If the site offers one, use it. It is faster, stable, permitted, and will not break when the layout changes. Scraping a site that publishes an API is choosing the worse tool.

What does robots.txt say? Visit example.com/robots.txt. It states which paths automated clients should not fetch. It is not a legal document, but ignoring it is a clear signal of bad faith and is often cited when access is challenged.

What do the terms of service say? Many sites prohibit automated collection outright. Scraping personal data brings data-protection obligations regardless of how public the page looked.

Practical rule for students: scraping public non-personal data for a college project, gently, is normally fine. Scraping personal details, or hammering a small site, is not — and the second one is visible to the site owner.

Fetching the page

import requests

headers = {"User-Agent": "Mozilla/5.0 (compatible; StudentProject/1.0)"}
r = requests.get("https://example.com/products", headers=headers, timeout=10)
r.raise_for_status()
html = r.text

Three details that are not optional in practice. timeout prevents the request hanging indefinitely — without it a stalled server hangs your script forever. raise_for_status() turns a 404 or 500 into an exception instead of letting you parse an error page as if it were data. And a descriptive User-Agent is basic courtesy; some sites reject the default one outright.

If you get a 403, that is usually the site declining automated access. Adding headers to disguise your scraper is possible and is a decision to make deliberately, not casually.

Parsing with BeautifulSoup

from bs4 import BeautifulSoup

html = """<div class="product"><h2>Laptop</h2>
            <span class="price">45000</span></div>
           <div class="product"><h2>Mouse</h2>
            <span class="price">500</span></div>"""

soup = BeautifulSoup(html, "html.parser")

for p in soup.select("div.product"):
    name = p.find("h2").text
    price = p.find("span", class_="price").text
    print(name, "->", price)

Laptop -> 45000
Mouse -> 500

select() takes CSS selectors, which are usually the most readable option. find() and find_all() take tag names and attributes — note class_ with the underscore, since class is a Python keyword.

Use the browser's element inspector to work out selectors. Prefer stable-looking hooks such as an id or a semantic class over generated names like css-1x2y3z, which change on every deployment.

When the page is empty

A common frustration: the data is clearly visible in the browser, and your scraper finds nothing.

That means the content is rendered by JavaScript after load. requests fetches the initial HTML only — it does not run scripts, so you are seeing what the browser saw before React or Vue populated the page.

Two options. First, look for the underlying API: open the browser's network tab, reload, and filter for XHR requests. Very often the page is calling a JSON endpoint you can call directly, which is faster and far more stable than parsing HTML. This is worth checking every time.

Second, use a browser automation tool such as Selenium or Playwright, which runs a real browser and executes the scripts. Much heavier and slower, and the right answer only when there is genuinely no underlying endpoint.

Not getting blocked, and not deserving to be

• Rate limit yourself. A time.sleep(1) between requests is the single most important line in a scraper. Hundreds of rapid requests look like an attack and can genuinely affect a small site.
• Cache while developing. Save the HTML to a file and iterate on your parsing against that. There is no reason to re-fetch the page fifty times while fixing a selector.
• Handle missing elements. p.find("h2") returns None when absent, and .text on None raises AttributeError. One malformed row should not kill the run.
• Expect breakage. Scrapers break when sites redesign. Write the parsing in one place so there is one thing to fix.
• Store raw and parsed separately. Keeping the raw HTML lets you re-parse without re-fetching when you discover a field you missed.

For the requests side in more depth, see the requests library.

Frequently Asked Questions