pathlib & file processing

Why file paths break in production

The bug that ruins a data pipeline at 2am is rarely the algorithm. It is a path. A script that reads data/scores.txt works on your laptop and fails on the server because the two machines were launched from different directories. Hardcoded string paths are also fragile across operating systems, where the path separator differs. pathlib.Path exists to make paths a real object with methods instead of fragile strings you glue together by hand. As a data engineer you touch files constantly (CSVs, logs, exports), so getting this layer right is table stakes.

The mental model

A Path is an object that represents a location, not the file's contents. Building one does no I/O and does not require the file to exist. You only touch disk when you call a method like read_text() or exists().

from pathlib import Path

p = Path("data") / "scores.txt"   # "/" joins path parts, OS-correct
p.exists()                        # True / False, cheap check
p.suffix                          # ".txt"
text = p.read_text(encoding="utf-8")   # whole file -> one str

The / operator is real: Path overloads it so Path("data") / "scores.txt" builds the joined path with the correct separator on any OS. Prefer it over "data/" + name.

Once you have a Path, its parts are attributes rather than string surgery:

Table

Two of these bite. .suffix keeps the dot, so comparing it to 'csv' always fails and you want '.csv'. And on a double extension like archive.tar.gz, .suffix is only '.gz' while .stem is still 'archive.tar', because both look at the LAST dot only.

Attribute	Value for Path('data/reports/scores.csv')	What you reach for it for
.name	scores.csv	the filename, extension included
.stem	scores	the filename without the extension
.suffix	.csv	the extension, INCLUDING the leading dot
.parent	data/reports	the containing directory, itself a Path
.parts	('data', 'reports', 'scores.csv')	every segment as a tuple

Turning text into data

read_text() hands you the entire file as one string. Split it into lines, then convert:

text = "10\n20\n30"
numbers = [int(line) for line in text.splitlines() if line.strip()]
print(sum(numbers))   # 60

splitlines() breaks on line boundaries and drops the \n characters. The if line.strip() guard skips blank or whitespace-only lines so int() never receives an empty string. int() itself strips surrounding whitespace, so int(" 10 ") returns 10 without extra work. This is exactly the shape both exercises want: first sum the numbers in a string, then read a real file with Path(path).read_text() and run the same pipeline.

Pitfalls

Real files almost always end with a trailing newline, and that is where interns get burned. Compare:

"10\n20\n30\n".split("\n")     # ['10', '20', '30', '']  <- trailing ''
"10\n20\n30\n".splitlines()    # ['10', '20', '30']       <- clean

If you use split("\n") you get a phantom empty string at the end, and int("") raises ValueError. Use splitlines(), or keep the if line.strip() guard, or both. That guard is your safety net for blank lines anywhere in the file, not just the last one.

The second trap: a relative path like Path("data/scores.txt") resolves against the current working directory, which is wherever the process was launched, not where your script file lives. Run the same code from a different folder and it fails. When it matters, anchor to the file with Path(__file__).parent / "data" / "scores.txt".

Interview nuance: read_text() loads the whole file into memory at once, so its memory cost is O(n) in file size. That is fine for a scores file, but if an interviewer swaps in a multi-gigabyte log, stream it instead and keep memory O(1):

with Path(path).open(encoding="utf-8") as f:
    total = sum(int(line) for line in f if line.strip())

Iterating a file object yields one line at a time without holding the whole file in memory. Knowing when to read all versus stream is exactly the tradeoff data-engineering interviewers probe.