Skip to main content
CodeSparring

Text is bytes underneath

Level 2: Level 2: Idiomsmedium12 minbytesencodingunicodedecoding

Encode and decode deliberately, and read the error that says these bytes were never UTF-8.

The first real wall you hit with real data

Toy data is ASCII, so text feels like a solved problem. Then a partner sends a customer list, one name has an accent in it, and the ingest dies with UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 0. Nothing in the code changed. The bytes changed, and the code had been quietly assuming what those bytes meant.

A file does not contain text. It contains bytes. Text is what you get when you interpret those bytes with a codec, and the whole family of encoding bugs comes from doing that interpretation by accident instead of on purpose.

str and bytes are different types

Table
Every file, socket, and HTTP body carries bytes. A str only ever exists inside your program, so encode and decode are the two doors between the outside world and it. Every text bug lives at one of those two doors.
Questionstrbytes
What it holdscharactersraw byte values, 0 to 255
How you write a literal'hi'b'hi'
What len() countscharactersbytes
How you reach the other one.encode('utf-8').decode('utf-8')
What a file or socket carriesnever thisalways this
Every file, socket, and HTTP body carries bytes. A str only ever exists inside your program, so encode and decode are the two doors between the outside world and it. Every text bug lives at one of those two doors.
text = "café"
raw = text.encode("utf-8")   # b'caf\xc3\xa9'
raw.decode("utf-8")          # back to 'café'

Read \xc3\xa9 as "two bytes that no ASCII character claims". UTF-8 spends one byte on every ASCII character and two to four bytes on everything else, which is exactly why it took over: plain English text costs nothing extra, and the rest of the world still fits.

Check yourself
text = 'cafe' with an accented final e, so four characters. What do len(text) and len(text.encode('utf-8')) return?

Why UnicodeDecodeError happens

UTF-8 is a structured encoding: a byte in the 0x80 to 0xFF range announces a multi-byte sequence and the bytes that follow have to fit the pattern. Bytes written by a different codec usually do not, so the decode fails:

raw = bytes([0xe9])          # 'é' as latin-1 wrote it
raw.decode("utf-8")          # UnicodeDecodeError: can't decode byte 0xe9 in position 0
raw.decode("latin-1")        # 'é'

The error message is unusually good. It names the codec it tried, the offending byte, and the position, which together tell you both what the data probably is and where to look.

Check yourself
The same bytes are decoded twice, once as utf-8 and once as latin-1, and they are not valid UTF-8. Which call raises?

errors="replace" buys quiet, and it costs data

Every decode takes an errors argument. The default is "strict", which raises. "replace" substitutes U+FFFD (the black diamond question mark) for anything it cannot read:

bytes([0xe9]).decode("utf-8", errors="replace")   # '\ufffd'
Check yourself
An ingest keeps failing on a few bad bytes, so someone changes the call to raw.decode('utf-8', errors='replace'). What does that buy, and what does it cost?

Use "replace" when a human is going to eyeball the output and a few garbled characters are survivable. Never use it on data you will store, compare, bill against, or send back to a customer, because the replacement is permanent and the original bytes are unrecoverable from that point on.

Always pass encoding= when you open a file

with open("names.csv", encoding="utf-8") as fh:
    text = fh.read()
Check yourself
open(path) with no encoding argument reads a file holding an accented name. It passes on the author's laptop and raises UnicodeDecodeError in production. Why?

Pitfalls

  • str and bytes never mix. "a" + b"b" raises TypeError, and "abc" == b"abc" is False. Decide which side of the boundary you are on and stay there.
  • Encoding can fail too. "café".encode("ascii") raises UnicodeEncodeError, which is the same problem running the other way.
  • A BOM is real bytes. A file saved by some Windows tools starts with \xef\xbb\xbf, and UTF-8 decodes it as an invisible character that then breaks your first column name. encoding="utf-8-sig" strips it.

Interview nuance: the sentence to have ready is "decode at the boundary, work in str, encode on the way out." It is the same shape as the naive-versus-aware rule for datetimes, and for the same reason: a program is easiest to reason about when every value inside it is already normalised. If pressed on a real incident, name latin-1 as the codec that never raises, so the failure it causes is wrong characters in your database rather than an exception in your logs.

Worked example (Python)
text = "café"
raw = text.encode("utf-8")
print(raw)                      # b'caf\xc3\xa9'
print(len(text), len(raw))      # 4 5
print(raw.decode("utf-8"))      # café

latin = bytes([0xe9])
print(latin.decode("latin-1"))                      # é
print(latin.decode("utf-8", errors="replace"))      # the replacement character

Apply

Your turn

The task this lesson builds to.

Write a function utf8_byte_values(text) that returns the list of byte values UTF-8 uses to store text.

For "hi" return [104, 105]. For a single accented character it will be two values, not one.

3 hints and 4 automated checks are waiting in the workspace.

Practice

Make it stick

A second problem on the same idea, so it survives past today.

A partner drops a nightly export onto your SFTP box. Most nights it is UTF-8 and the ingest is quiet. Then one file is written by an older Windows tool in latin-1, a single accented surname lands in it, and the whole batch dies on UnicodeDecodeError. You cannot make the partner change, so the loader has to cope.

Implement decode_with_fallback(byte_values): build bytes from the list of byte values, decode it as UTF-8, and when that raises UnicodeDecodeError, decode it as latin-1 instead. Return the resulting str either way.

3 hints and 4 automated checks are waiting in the workspace.