Skip to content
AyoKoding

Overview

Prerequisites

  • Prior topics: 4 · Just Enough Python -- every script in this topic is fully type-annotated Python, and you should already be comfortable reading functions, classes, exceptions, and list/dict literals the way that primer taught them; 5 · Just Enough Bash -- every example is driven from a terminal, and several examples pipe commands, background a process with &, or read an exit code the way that primer taught; 15 · Software Testing -- this topic assumes you can already write a failing pytest-style assertion and read a traceback; a test tells you that something is wrong, this topic is about finding where and why, then measuring whether a fix actually helped.
  • Tools & environment: a macOS/Linux terminal; Python 3.13.12 (stdlib pdb, cProfile, pstats, tracemalloc, threading, asyncio, sys.monitoring -- no extra install needed) plus py-spy 0.4.2, debugpy 1.8.21, line_profiler 5.0.2 (provides kernprof), hypothesis 6.156.6, gprof2dot 2025.4.14, and snakeviz 2.2.2 from PyPI; Python 3.14.3 for the two examples that need pdb's 3.14-only remote-attach and set_trace_async features; git 2.39.5+ for the bisection examples; inferno (cargo install inferno) for flame-graph rendering; on Linux, gdb 17.2+ (bundles Python-awareness via python-gdb.py) and perf (ships with the Linux kernel's linux-tools package) for the native-debugging tier; on macOS, lldb (ships with Xcode Command Line Tools) plus cpython_lldb 0.3.3 from PyPI.
  • Assumed knowledge: reading and writing typed Python functions and classes; running a program and reading its exit code from a Bash terminal; writing a plain assert-based test. No prior debugging or profiling background is required -- this topic is where that background starts.

Why this exists -- the big idea

The problem before the solution: a failing test tells you something is wrong but not where or why; print-driven guessing scales poorly as a codebase grows, and optimizing by hunch tunes the wrong 90% of the code while the real bottleneck sits untouched, unmeasured. The one idea worth keeping if you forget everything else: debugging is a search -- form a falsifiable hypothesis, change exactly one thing, observe, and halve the remaining search space; performance work is measure-first, because the hot spot is almost never where you would have guessed by reading the code.

Cross-cutting big ideas, taught here and then reused for the rest of this topic: layering-and-leaks -- a bug or a hot spot usually lives at a seam between layers (your code, the runtime, the OS, the CPU cache), and a profiler or a native-aware debugger is how you see through that layer instead of guessing at it (the Native & Systems tier makes this literal: the same slow function looks completely different from cProfile's Python-only view versus a native-aware view). determinism-vs-emergence -- the hardest bugs are emergent, not local: a data race or an asyncio interleaving bug depends on the interaction between threads or tasks, not on any single line, so it is only reproducible by controlling that interaction (a forced yield, a seed, a barrier), never merely by rereading the code closely enough.

Install and run your first example

Confirm the core toolchain this topic's Beginner tier uses is installed:

$ python3 --version
Python 3.13.12
$ python3 -m pdb --help > /dev/null && echo "pdb OK"
pdb OK
$ python3 -c "import cProfile, pstats, tracemalloc; print('stdlib profiling OK')"
stdlib profiling OK
$ git --version
git version 2.39.5

Install the packages the Intermediate/Advanced/Native & Systems tiers need with one pip command (ideally into a virtual environment):

pip install py-spy==0.4.2 debugpy==1.8.21 line_profiler==5.0.2 hypothesis==6.156.6 \
  gprof2dot==2025.4.14 snakeviz==2.2.2

A note on versions: this topic's examples were authored and verified against these exact tool versions, in this sandbox, on 2026-07-15. pdb, cProfile, tracemalloc, and git bisect are stdlib/core-tool features that have been stable for many releases; the two examples that need Python 3.14's remote-attach (python -m pdb -p <pid>) and pdb.set_trace_async() are called out explicitly where they appear.

Every example is a complete, self-contained runnable file (or command sequence) colocated under learning/code/, actually executed to capture its documented output. Where an example genuinely needs a privilege or a host this sandbox does not have (root access for py-spy on macOS, a Linux kernel for perf, macOS Developer Mode for lldb attach), that limitation is stated honestly, backed by the tool's own error message or documentation, rather than a fabricated transcript standing in for it -- see the Native & Systems tier's opening note for the full, itemized account of every such limitation and its real, disclosed substitute.

How this topic's examples are organized

  • Beginner (Examples 1-27) -- pdb's core vocabulary: breakpoints, stepping, the call stack, conditional breaks, display, post-mortem debugging, print/logging versus interactive debugging, plus a first pass through cProfile, tracemalloc, flame-graph reading, and git bisect by hand.
  • Intermediate (Examples 28-49) -- py-spy's top/record/dump (and the honest substitute used where root access is unavailable), sorting pstats by tottime versus cumtime, line-level profiling with kernprof, tracemalloc diffing, debugpy/DAP remote attach, automated git bisect run, and delta-debugging with a hand-rolled ddmin and with Hypothesis's shrinker.
  • Advanced (Examples 50-62) -- pdb's interact mode and chained-exception exceptions command, multi-module log correlation, converting a profile to a flame graph via gprof2dot, a real Neovim DAP session, a documented before/after cProfile measurement, a genuine threading race reproduced and fixed, an asyncio interleaving bug, pdb.set_trace_async(), multiprocessing versus threading under load, automated bisection of a performance regression, and delta-debugging a 10,000-line crash.
  • Native & Systems (Examples 63-80) -- gdb/lldb/perf-level native-aware debugging and profiling of the CPython process itself (with every host limitation disclosed honestly), plus the topic's integrative discipline examples: a combined correctness-and-performance bug, the recursive tottime-versus-cumtime trap, an unbounded-cache leak, import-time profiling, lock contention under load, a flame-graph diff, deterministic seeding for a flaky bug, bisecting with a flaky-test guard, and a low-overhead sys.monitoring tracer.
  • Capstone -- one repository carrying both a seeded correctness bug and a seeded performance bug, resolved end to end: bisect and minimize the failing input, fix it with a debugger-guided, test-first change, then profile (two independent ways), fix, and measure the hot path with zero regressions.

The 23 concepts this topic covers

  • co-01 · Interactive breakpoints and stepping -- pausing execution at a chosen line and driving it forward with step-into / step-over / step-out, versus letting it run to completion. Examples 1-3, 14, 25-26, 39, 54, 59.
  • co-02 · Conditional and watch breakpoints -- halting only when a predicate holds (i == 47, id(obj) == ...) or a value changes, instead of stopping on every hit. Examples 8-11, 38-39.
  • co-03 · Call-stack, frame, and variable inspection -- reading the stack, moving between frames, and printing or mutating locals at a paused point to test a hypothesis. Examples 4-7, 25, 32, 50-51, 64.
  • co-04 · Post-mortem debugging -- entering a debugger on an already-thrown, uncaught exception (pdb.pm(), python -m pdb) to inspect the failing frame without a rerun. Examples 15-16, 42, 51, 66, 72.
  • co-05 · Print/logging versus interactive debugging -- when print/logging beats a breakpoint (long-lived processes, production, timing) and when it does not (one-off, deep state). Examples 12-13, 52.
  • co-06 · Remote and DAP debugging -- attaching a debugger to an already-running process over the Debug Adapter Protocol (debugpy, editor DAP) or pdb -p <pid>. Examples 40-42, 54, 59.
  • co-07 · Scientific-method debugging loop -- expected-vs-actual, one falsifiable hypothesis, one change, observe, repeat -- debugging as controlled experiment, not guessing. Examples 7, 23, 50, 78.
  • co-08 · Bisection search as a general strategy -- halving the search space (commits, input, code region) to localize a fault in logarithmic rather than linear steps. Examples 22, 24, 43.
  • co-09 · git bisect, manual -- driving git bisect start/good/bad by hand to name the commit that introduced a regression. Examples 22, 72.
  • co-10 · git bisect, automated -- git bisect run <script> with a pass/fail (or 125-skip) exit code so the whole search runs unattended. Examples 43-44, 61, 72, 79.
  • co-11 · Delta-debugging input minimization -- shrinking a failing input to a 1-minimal reproducer (hand-rolled ddmin, Hypothesis shrinking) so every remaining piece is necessary. Examples 24, 45-47, 62.
  • co-12 · Sampling versus instrumenting profilers -- periodic stack sampling (low overhead, statistical) versus per-call instrumentation (exact counts, high overhead), and when each is right. Examples 28-29.
  • co-13 · CPU profiling with cProfile -- the stdlib instrumenting profiler: running it from the CLI or programmatically and reading its pstats table. Examples 17-18, 28, 33, 35, 53, 55, 70, 72-73, 75.
  • co-14 · CPU profiling with py-spy -- a sampling profiler that attaches to a live PID with no code changes (top, record, dump, --native) -- and the honest, disclosed substitute (mini_sampler.py, built from sys._current_frames()) used wherever root access is unavailable. Examples 28-32, 60, 71.
  • co-15 · Wall-clock versus CPU time -- distinguishing elapsed time (perf_counter) from on-CPU time (process_time); I/O-bound versus CPU-bound diagnosis. Examples 19, 60, 76.
  • co-16 · tottime versus cumtime -- a function's own time versus time including callees, and which one to optimize (many-cheap leaf versus one-expensive parent). Examples 18, 33, 73.
  • co-17 · Memory profiling with tracemalloc -- snapshotting allocations, diffing two snapshots to find a leak, and widening the traceback (nframe) to split allocation sites. Examples 20, 36-37, 74.
  • co-18 · Line-level profiling -- attributing cost to individual source lines (line_profiler's @profile plus kernprof), one resolution finer than function-level. Examples 34-35.
  • co-19 · Flame-graph reading -- width = total time in a call subtree, height = stack depth; finding the widest frame (the hot spot), not the tallest stack. Examples 21, 30-31, 53, 68-69, 71, 77.
  • co-20 · Race and heisenbug reproduction -- forcing nondeterministic concurrency bugs to reproduce reliably (forced yields, seeds, barriers) before attempting a fix. Examples 56-58, 76, 78-79.
  • co-21 · Load-representative versus toy profiling -- profiling at realistic scale and concurrency, because the hot spot at 100 items or a single call differs from production. Examples 48-49, 60, 76.
  • co-22 · Native-layer costs and native debugging -- seeing costs the interpreter hides: gdb/lldb Python-aware backtraces, perf with Python perf-maps, py-spy --native, core dumps. Examples 63-68, 70-71, 75, 80.
  • co-23 · Before/after measurement discipline -- proving a fix with a repeated, documented measurement, and confirming zero regressions. Examples 27, 55, 57, 72, 74-75, 77, 80.

Examples by Level

Beginner (Examples 1–27)

Intermediate (Examples 28–49)

Advanced (Examples 50–62)

Native & Systems (Examples 63–80)

← Previous: Overview · Next: Beginner Examples →

Last updated July 14, 2026

Command Palette

Search for a command to run...