Fetching JSON from an API

Level 2: Level 2: Idiomsmedium13 minhttpapisjsonerror-handling

Read the shape of an HTTP call, then handle the half that actually breaks: the status and the parse.

One line fetches. The other ten decide what to do about it

Calling an API is three steps: send a request, look at what came back, and turn the body into Python values. The sending is one line and it is the part nobody gets wrong. Everything that breaks in production lives in the other two steps, because a response can arrive perfectly well and still be a failure, and a body can arrive perfectly well and still not be the shape you expected.

This editor runs Python in your browser, which has no network, so nothing here will actually fetch. Read the call below to learn its shape, then practise the half that breaks: reading a status and parsing a body. Level 3 does the rigorous version against a real service with httpx and pydantic.

The shape of the call

import httpx

response = httpx.get("https://api.example.com/repos/python/cpython", timeout=5.0)
response.raise_for_status()      # turn a 4xx or 5xx into an exception, here
payload = response.json()        # dict, list, str, int, bool or None
payload["name"]                  # 'cpython'

Four things are doing work in those lines. timeout stops a hung server from hanging your process too. status_code carries the verdict. raise_for_status() converts a bad status into an exception at a line you chose, rather than letting it travel on as data. And .json() parses the body, which is the same json.loads you already know, run over whatever bytes came back.

Check yourself

httpx.get(url) comes back with status 404 and an HTML error page. Your next line is response.json(). What did the get call itself do?

It raised, so the next line never runsIt returned normally. The 404 is just the status on an ordinary response objectIt returned None, since there is no useful body to parseIt retried automatically until the URL resolved

The status is a decision, not a detail

Table

The split that matters is transient versus permanent. 429 and 5xx can clear on their own, so a retry is useful. Other 4xx statuses say the request itself is wrong, so an identical retry produces an identical failure while spending your rate limit.
Status	What it means	What the caller should do
2xx	the request worked	parse the body
429	you are being rate limited	wait, then retry, honouring Retry-After
other 4xx	your request is wrong	fix the request; retrying changes nothing
5xx	the other side is broken	retry with backoff, then give up loudly
no response at all	a timeout or a connection error	this is the case that raises, so catch it

The split that matters is transient versus permanent. 429 and 5xx can clear on their own, so a retry is useful. Other 4xx statuses say the request itself is wrong, so an identical retry produces an identical failure while spending your rate limit.

Check yourself

A worker retries every failed call five times with a backoff. Which status is the one where retrying is genuinely useful?

400, because a bad request might go through on a second attempt503, because the service is temporarily unavailable401, because the token may be accepted next time404, because the resource may show up shortly

Check yourself

A load balancer returns an HTML maintenance page with status 502. The code calls response.json() without checking the status first. What comes back?

None, because the body is not JSONA JSON decode error, and the traceback blames your parsing rather than the 502The HTML as a plain string, since json() falls back to textAn empty dict, which the code then reads as a missing record

Parse the body as if it were written by a stranger

Because it was. A response is untrusted input: a field can be absent, be null, be a string where you expected a number, or be an empty list where you expected one element. Chained square brackets assume all of it is fine:

payload["owner"]["login"]          # KeyError the first time owner is missing
payload["items"][0]["id"]          # IndexError the first time a page is empty

owner = payload.get("owner") or {}
owner.get("login", "unknown")      # a default you chose, at the field you chose

Check yourself

A response usually looks like a dict with an owner key holding a dict with a login key. One record has no owner key at all, and the code reads payload['owner']['login']. What happens?

It returns None, which the next line can handleKeyError naming 'owner', which kills the whole batch on one bad recordKeyError naming 'login', since the outer lookup succeededTypeError, because you cannot index into a missing value

Note that .get("owner") returns None both when the key is missing and when the value really is JSON null, so payload.get("owner").get("login") still raises AttributeError on a null. The or {} above is what closes that gap, and the Apply exercise makes you handle exactly this case.

Pitfalls

No timeout means no limit. httpx defaults to five seconds, but requests has no default at all, so a hung server can pin a worker forever. Always pass one explicitly and you never have to remember which library you are in.
Retrying without backoff is a denial of service you aimed at yourself. Sleep longer after every attempt, cap the number of attempts, and honour Retry-After when a 429 sends one.
A 200 does not mean the body is right. Plenty of APIs return errors inside a 200 envelope. Check the payload's own success field where one exists.

Interview nuance: when you are asked to "call this API," the answer that lands is the failure taxonomy, not the request line. Say that you separate three cases: no response at all (a timeout or connection error, and the only case that raises), a response with a bad status (transient for 429 and 5xx, so retry with backoff; permanent for other 4xx, so fail loudly), and a response with a good status but an unexpected shape (validate at the boundary and reject the record, not the batch). Level 3 turns that third case into typed models with pydantic; the reasoning is the same at every scale.

Worked example (Python)

# No network here, so this is what a real response would already have handed you.
payload = {
    "name": "cpython",
    "owner": {"login": "python", "id": 1},
    "items": [],
}

print(payload["owner"]["login"])              # python
print((payload.get("license") or {}).get("key", "none"))   # none

for status in (200, 404, 429, 503):
    if 200 <= status < 300:
        print(status, "ok")
    elif status == 429 or 500 <= status < 600:
        print(status, "retry")
    else:
        print(status, "fail")

Apply

Your turn

The task this lesson builds to.

Write a function owner_login(payload) that returns the login name nested at owner.login in an API response, or the string "unknown" when the response does not carry one.

{"name": "cpython", "owner": {"login": "python"}} returns "python". A payload with no owner key, with an empty owner, or with owner set to null all return "unknown" instead of raising.

3 hints and 4 automated checks are waiting in the workspace.

Practice

Make it stick

A second problem on the same idea, so it survives past today.

An ingest worker polls a vendor's API every five minutes. Some failures are the vendor's and clear on their own, and some are the worker's own fault and will fail identically forever. Retrying the second kind burns the rate limit and hides the bug, so the worker has to tell them apart before it decides anything.

Implement retry_decision(status): return "ok" for any 2xx status, "retry" for 429 and for any 5xx, and "fail" for everything else.

3 hints and 4 automated checks are waiting in the workspace.