Working with APIs

What an API Call Really Is

An API is a way for one program to ask another for data or action. On the web, that conversation is HTTP: your program sends a request and the server sends back a response. Everything else in this lesson is detail about those two messages.

A request has four parts. The URL says what you want. The method says what kind of operation it is — GET to read, POST to create, PUT or PATCH to update, DELETE to remove. The headers carry metadata such as who you are and what format you can accept. The body carries data you are sending, and is empty for a GET.

A response has three: a status code saying how it went, headers, and a body. For the APIs you will meet, that body is almost always JSON — the format from Lesson 16 — which maps cleanly onto Python dictionaries and lists.

Python's standard library can do this with urllib.request, and almost nobody does. The third-party requests library turns a page of setup into one line, handles encoding and redirects sensibly, and is what every tutorial, job posting and colleague assumes. Install it with python -m pip install requests, inside a virtual environment.

One habit before you start: read the API's documentation first. It tells you the base URL, which endpoints exist, what authentication is needed, what the rate limit is, and what the response looks like. Ten minutes there saves an hour of guessing from error messages.

import requests

# The simplest possible call
response = requests.get("https://jsonplaceholder.typicode.com/posts/1")

print(response.status_code)      # 200
print(response.headers["Content-Type"])   # application/json; charset=utf-8

data = response.json()           # JSON body -> a Python dict
print(type(data))                # <class 'dict'>
print(data["title"])

# The raw text, if you need to see exactly what arrived
print(response.text[:60])

# The request that was actually sent
print(response.request.method)   # GET
print(response.url)

# The four methods you will use
# requests.get(url)                 read
# requests.post(url, json=data)     create
# requests.put(url, json=data)      replace
# requests.patch(url, json=data)    update part
# requests.delete(url)              remove

• Request: URL, method, headers, body
• Response: status code, headers, body (usually JSON)
• GET read, POST create, PUT/PATCH update, DELETE remove
• response.json() — parse the body into Python dicts and lists
• response.text — the raw body as a string; response.content for bytes
• requests is third-party: python -m pip install requests

Query Parameters, Timeouts and Reading the Response

Most GET endpoints accept filters as query parameters — the part of a URL after the question mark. Pass them as a dictionary to params= and let requests build the URL. Do not build it by string concatenation: values containing spaces, ampersands or non-English characters have to be percent-encoded, and getting that wrong produces requests that fail in ways that are painful to diagnose.

Now the most important argument in this lesson: always pass timeout=. By default requests waits forever. If the server accepts your connection and then goes quiet, your program hangs with no error, no traceback and no clue — and that is precisely how a scheduled script silently stops producing reports. A few seconds for a fast API, longer for one you know is slow, but never nothing.

Reading the response is straightforward once it arrives. .json() parses the body; .text gives it as a string; .content gives raw bytes, which is what you want for an image or a PDF. .headers is a dictionary of metadata, and it is worth looking at — that is where rate-limit counters and pagination links usually live.

The one trap when reading is that .json() assumes the body is JSON. When something has gone wrong, servers frequently return an HTML error page with a perfectly ordinary content type, and .json() then raises a decoding error that mentions nothing about the real problem. Check the status code first, and the confusing failure never happens.

import requests

BASE = "https://jsonplaceholder.typicode.com"

# Let requests build the query string — it handles encoding
r = requests.get(f"{BASE}/posts", params={"userId": 1}, timeout=10)
print(r.url)                 # .../posts?userId=1
print(len(r.json()))         # how many posts came back

# Values that need encoding are handled for you
r = requests.get(f"{BASE}/posts", params={"q": "data science & AI"},
                 timeout=10)
print(r.url)                 # ...?q=data+science+%26+AI

# ALWAYS set a timeout — the default is to wait forever
try:
    r = requests.get(f"{BASE}/posts/1", timeout=5)
except requests.exceptions.Timeout:
    print("the server did not answer in 5 seconds")

# Reading what came back
print(r.status_code, r.ok)             # 200 True
print(r.headers.get("Content-Type"))
data = r.json()
print(data["title"][:40])

# Check the status BEFORE parsing
r = requests.get(f"{BASE}/nope", timeout=5)
if r.ok:
    print(r.json())
else:
    print(f"{r.status_code}: {r.text[:80]}")   # no confusing JSON error

# Binary bodies use .content, not .text
img = requests.get("https://www.python.org/static/img/python-logo.png",
                   timeout=10)
with open("logo.png", "wb") as f:
    f.write(img.content)

• params={...} — query parameters, correctly encoded for you
• timeout=n — never omit it; the default is to wait indefinitely
• response.ok — True for any status below 400
• .json() / .text / .content — parsed, string, bytes
• response.headers — rate limits and pagination links often live here
• Check the status before calling .json()

Status Codes and What They Are Telling You

The status code is the server's summary of what happened, and the first digit is the part to learn. 2xx means success. 3xx means the resource moved, which requests follows automatically. 4xx means you did something wrong. 5xx means the server did.

The distinction between 4xx and 5xx is the one that decides what your program should do. A 4xx will fail again identically no matter how many times you retry — fix the request instead. A 5xx is often temporary, so retrying after a pause is reasonable.

The individual codes worth recognising: 200 fine, 201 created, 204 done with nothing to return. 400 your request was malformed, 401 you are not authenticated, 403 you are authenticated but not allowed, 404 no such thing, 429 you are sending too fast. 500 the server crashed, 503 it is temporarily unavailable.

The 401-versus-403 difference confuses people and is genuinely useful: 401 means "I do not know who you are" — check your key. 403 means "I know who you are and you may not do this" — check your permissions. Sending a different key will not help with a 403.

response.raise_for_status() turns any 4xx or 5xx into an exception, which is the clean way to stop a failure from flowing onward as data. Use it when a failure genuinely should stop the operation, and check the code by hand when different failures need different handling — 404 meaning "not found, that is fine" is a common example.

import requests

BASE = "https://jsonplaceholder.typicode.com"

r = requests.get(f"{BASE}/posts/999999", timeout=10)
print(r.status_code)          # 404
print(r.ok)                   # False

# raise_for_status: turn a bad status into an exception
try:
    r.raise_for_status()
except requests.exceptions.HTTPError as e:
    print("failed:", e)       # 404 Client Error: Not Found for url: ...

# When different codes need different handling
def fetch_post(post_id):
    r = requests.get(f"{BASE}/posts/{post_id}", timeout=10)

    if r.status_code == 404:
        return None                       # missing is not an error here
    if r.status_code == 401:
        raise PermissionError("API key missing or wrong")
    if r.status_code == 403:
        raise PermissionError("key is valid but lacks permission")
    if r.status_code == 429:
        wait = int(r.headers.get("Retry-After", 60))
        raise RuntimeError(f"rate limited; retry in {wait}s")

    r.raise_for_status()                  # anything else 4xx/5xx
    return r.json()

print(fetch_post(1)["title"][:30])
print(fetch_post(999999))                 # None

# 4xx: retrying will not help.  5xx: it might.
for code in (400, 404, 429, 500, 503):
    kind = "your request" if code < 500 else "the server"
    print(code, "->", kind)

• 2xx success — 200 OK, 201 Created, 204 No Content
• 4xx your fault — 400 bad request, 401 not authenticated, 403 forbidden, 404 not found, 429 too many requests
• 5xx the server's fault — 500 error, 503 unavailable
• Retry 5xx and 429; never retry a 4xx unchanged
• raise_for_status() — turn 4xx and 5xx into an HTTPError
• 401 means "who are you"; 403 means "not allowed" — different fixes

Sending Data: json= versus data=

To create or update something you send a body, and the argument you choose decides its format. json=payload serialises a Python dictionary to JSON and sets the Content-Type header to application/json for you. data=payload sends it form-encoded, the format an HTML form uses.

Getting this wrong is a common source of a 400 Bad Request that looks inexplicable. An API expecting JSON receives form-encoded text, cannot parse it, and rejects the call. Modern REST APIs almost always want json=; use data= only when the documentation asks for form encoding, which is most common in older login endpoints.

You can also pass a pre-serialised string to data=, but then you must set the header yourself. There is no reason to do this unless an API needs an unusual format — json= does both jobs correctly.

The other methods follow the same shape. PUT replaces a resource entirely, so send every field. PATCH updates part of one, so send only what changes. DELETE usually needs no body and often answers 204 No Content, which means calling .json() on it will fail — there is nothing there to parse.

After a successful POST, read the response body. APIs normally return the created object including the identifier the server assigned, and that identifier is what you need for every later call about it.

import requests

BASE = "https://jsonplaceholder.typicode.com"

new_post = {"title": "Learning Python", "body": "Notes from Lesson 23",
            "userId": 1}

# JSON body — what modern REST APIs expect
r = requests.post(f"{BASE}/posts", json=new_post, timeout=10)
print(r.status_code)                 # 201 Created
created = r.json()
print(created["id"])                 # the id the server assigned
print(r.request.headers["Content-Type"])   # application/json

# Form-encoded body — only when the API asks for it
r = requests.post(f"{BASE}/posts", data=new_post, timeout=10)
print(r.request.headers["Content-Type"])
# application/x-www-form-urlencoded

# PUT replaces the whole resource
r = requests.put(f"{BASE}/posts/1",
                 json={"id": 1, "title": "Updated", "body": "All fields",
                       "userId": 1},
                 timeout=10)
print(r.status_code)                 # 200

# PATCH updates part of it
r = requests.patch(f"{BASE}/posts/1", json={"title": "Just the title"},
                   timeout=10)
print(r.json()["title"])

# DELETE often returns no body at all
r = requests.delete(f"{BASE}/posts/1", timeout=10)
print(r.status_code)
if r.status_code != 204 and r.text.strip():
    print(r.json())                  # only parse when there is something

Authentication and Keeping Secrets Out of Your Code

Most useful APIs need to know who you are. The usual mechanism is a key or token sent in a header — commonly Authorization: Bearer YOUR_TOKEN, sometimes a custom header such as X-API-Key. A few older APIs accept the key as a query parameter, which is worse, because URLs end up in server logs and browser history.

The rule that matters more than the mechanism: never write a key in your source code. Not temporarily, not "just for testing". Code gets committed, pushed and shared, and a key in a public repository is found by automated scanners within minutes. Every year students lose access to accounts, and occasionally run up real charges, because a key went to GitHub inside a practice project.

Keep secrets in environment variables and read them with os.environ. For local development, a .env file loaded by python-dotenv is convenient — provided .env is listed in .gitignore, which is the whole point. Commit a .env.example with the variable names and no values, so the next person knows what to set.

Read the key at the top of your program and fail immediately with a clear message if it is missing. Discovering the absence of a key through a 401 halfway through a batch job is far less pleasant than a one-line error before anything starts.

If a key does get committed, treat it as compromised even after you delete the line. Git keeps history, so the value is still in the repository. Revoke the key and issue a new one — that is the only fix that actually works.

import os
import requests

# Read the secret from the environment; fail early and clearly
API_KEY = os.environ.get("WEATHER_API_KEY")
if not API_KEY:
    raise RuntimeError("set WEATHER_API_KEY before running this script")

# The usual place: a header
headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Accept": "application/json",
    "User-Agent": "priodemy-course-example/1.0",
}

# r = requests.get("https://api.example.com/v1/weather",
#                  params={"city": "Kolkata"},
#                  headers=headers, timeout=10)

# A Session keeps headers and connections across calls
session = requests.Session()
session.headers.update(headers)
# r = session.get("https://api.example.com/v1/weather", timeout=10)

# Never do this
# API_KEY = "sk_live_9f3a..."     # committed, pushed, scanned, stolen

# Setting the variable, per platform:
#   Windows PowerShell:  $env:WEATHER_API_KEY = "your-key"
#   macOS / Linux:       export WEATHER_API_KEY="your-key"

# Or a .env file loaded at startup (pip install python-dotenv)
# from dotenv import load_dotenv
# load_dotenv()                    # reads .env into os.environ
#
# .gitignore must contain:
#   .env
# and commit .env.example instead:
#   WEATHER_API_KEY=

• Keys usually go in a header: Authorization: Bearer TOKEN
• Never hardcode a secret — not even temporarily
• os.environ.get("NAME") — read it from the environment
• python-dotenv plus a .env file for local development, with .env in .gitignore
• Check for the key at startup and fail with a clear message
• A committed key stays in git history — revoke it, do not just delete the line

A Client That Survives Contact With Reality

Networks fail, servers restart, and rate limits exist. Code that assumes every call succeeds works perfectly in a demo and badly everywhere else. A few habits make the difference.

Wrap repeated calls in a small class. It holds the base URL, the session and the shared headers in one place, gives every endpoint a readable method name, and puts error handling in one place instead of at every call site. A requests.Session also reuses the underlying connection, which is measurably faster when you make many calls to the same host.

Handle the exceptions requests raises, not just bad status codes. Timeout means no answer in time; ConnectionError means the host could not be reached at all — no internet, wrong domain, DNS failure; HTTPError comes from raise_for_status(). All three inherit from RequestException, so catching that covers everything network-related.

Retry with a growing pause. Retrying immediately after a failure usually hits the same problem and adds load to a server that is already struggling. Waiting one second, then two, then four — exponential backoff — is the standard approach. Retry only 5xx and 429, never a 4xx, and give up after a fixed number of attempts.

Finally, handle pagination. APIs rarely return ten thousand records at once; they return a page and tell you how to get the next, either through a page number or a next URL in the body or headers. Writing that loop as a generator is a neat fit for Lesson 21: the caller iterates over items and never thinks about pages.

import time
import requests

class PostClient:
    """Small client for the JSONPlaceholder posts API."""

    BASE_URL = "https://jsonplaceholder.typicode.com"

    def __init__(self, timeout=10, retries=3):
        self.timeout = timeout
        self.retries = retries
        self.session = requests.Session()
        self.session.headers.update({"Accept": "application/json"})

    def _get(self, path, **params):
        url = f"{self.BASE_URL}{path}"
        delay = 1
        for attempt in range(1, self.retries + 1):
            try:
                r = self.session.get(url, params=params, timeout=self.timeout)
                if r.status_code == 429 or r.status_code >= 500:
                    raise requests.exceptions.HTTPError(r.status_code)
                r.raise_for_status()          # 4xx: do not retry
                return r.json()
            except (requests.exceptions.Timeout,
                    requests.exceptions.ConnectionError,
                    requests.exceptions.HTTPError) as e:
                if attempt == self.retries:
                    raise
                print(f"attempt {attempt} failed ({e}); waiting {delay}s")
                time.sleep(delay)
                delay *= 2                    # exponential backoff

    def get_post(self, post_id):
        return self._get(f"/posts/{post_id}")

    def posts_by_user(self, user_id):
        return self._get("/posts", userId=user_id)

    def iter_pages(self, path, per_page=20):
        """Yield items page by page, so the caller never sees pages."""
        page = 1
        while True:
            batch = self._get(path, _page=page, _limit=per_page)
            if not batch:
                return
            yield from batch
            page += 1


client = PostClient()
print(client.get_post(1)["title"][:40])
print(len(client.posts_by_user(1)), "posts")

from itertools import islice
for post in islice(client.iter_pages("/posts"), 3):
    print("-", post["title"][:40])

• One client class: base URL, session, headers and error handling in one place
• requests.Session() — reuses connections and persists headers
• Catch Timeout, ConnectionError and HTTPError, or their base RequestException
• Retry 5xx and 429 with exponential backoff; never retry a 4xx unchanged
• Paginate with a generator so callers iterate over items, not pages
• Respect documented rate limits — being blocked costs more time than waiting