The requests Library: Calling APIs Without the Sharp Edges

Quick Answer

requests.get() and requests.post() cover most needs. Always pass timeout, always call raise_for_status() or check the status code, and use params and json arguments rather than building URLs and bodies by hand.

The basic calls

import requests

r = requests.get("https://api.example.com/students", timeout=10)
print(r.status_code)      # 200
data = r.json()           # parsed JSON as a dict or list

Query parameters go in params, not concatenated into the URL — requests handles the encoding, which matters as soon as a value contains a space or an ampersand:

r = requests.get(url, params={"stream": "Science", "min_marks": 80}, timeout=10)
print(r.url)   # ...?stream=Science&min_marks=80

Sending JSON uses the json argument, which serialises the body and sets the Content-Type header for you:

r = requests.post(url, json={"name": "Asha", "marks": 91}, timeout=10)

Using data= with a manually stringified dictionary is the common mistake — it sends form encoding, and the server usually rejects it with a confusing 400.

timeout is not optional

Without it, requests waits indefinitely. If the server accepts the connection and then never responds, your program hangs forever with no error — no exception, no log line, nothing to debug.

This is the single most common cause of a scheduled job that silently stops running. It did not crash; it is still sitting there waiting.

r = requests.get(url, timeout=10)              # total seconds
r = requests.get(url, timeout=(3.0, 10.0))     # (connect, read)

Pass it on every single call. There is no default, and the absence of one is deliberate — the library cannot guess what is reasonable for your case.

Errors: two different kinds

Beginners check one and miss the other.

HTTP error responses — the request succeeded, the server replied 404 or 500. requests does not raise for these; r.json() will happily try to parse the error page:

r = requests.get(url, timeout=10)
r.raise_for_status()      # raises HTTPError on 4xx or 5xx

Network failures — DNS failure, connection refused, timeout. These raise exceptions:

try:
    r = requests.get(url, timeout=10)
    r.raise_for_status()
    data = r.json()
except requests.exceptions.Timeout:
    print("timed out")
except requests.exceptions.HTTPError as e:
    print("server returned", e.response.status_code)
except requests.exceptions.RequestException as e:
    print("request failed:", e)

RequestException is the base class, so catching it covers everything requests can raise. Note r.json() raises its own error if the body is not valid JSON — which is what happens when an API returns an HTML error page.

Headers, auth and sessions

headers = {"Authorization": f"Bearer {token}",
           "Accept": "application/json"}
r = requests.get(url, headers=headers, timeout=10)

Keep the token in an environment variable, never in the source. A committed API key is one of the most common secrets leaked from student repositories — see what recruiters look at on your GitHub.

For several calls to the same host, use a Session:

with requests.Session() as s:
    s.headers.update({"Authorization": f"Bearer {token}"})
    a = s.get(url1, timeout=10)
    b = s.get(url2, timeout=10)

A session reuses the underlying TCP connection, which avoids repeating the handshake on every call and is noticeably faster for many requests. It also carries cookies across them, which is what you need for anything session-based.

Practical habits

• Print the raw response when parsing fails. Most "the API is broken" reports turn out to be an HTML error page. print(r.status_code, r.text[:200]) resolves it immediately.
• Retry transient failures with backoff, not in a tight loop. Retrying a 500 immediately three times usually fails three times. Retrying a 400 is pointless — the request itself is wrong.
• Respect rate limits. A 429 response means you are going too fast, and it often includes a Retry-After header telling you exactly how long to wait.
• Stream large downloads with stream=True and write in chunks, rather than loading a large file entirely into memory.
• Do not disable certificate verification. verify=False appears in a lot of copied answers and removes the protection HTTPS exists to provide — see HTTP vs HTTPS.

Frequently Asked Questions