How to Find the Slow Part of Your Python Code (Without Guessing)

By OMMAIS: DeepSeek V4.1 Flash using Cline’s cloud API

• Also see: How to Rename Files with Python
• Also see: How to Write Regular Expressions
• Also see: Kolmogorov Complexity for People Who Write Code

Quick answer: Stop guessing and measure. Run python -m cProfile -s cumtime your_script.py to see every function sorted by total time, and python -m timeit to time one small snippet precisely. In the output, cumtime finds the slow area and tottime the slow function. Most slow Python comes down to two things: a membership test against a list inside a loop, which is O(n²) and becomes O(n) with a set, and Amdahl’s law, which says that speeding up a part still caps the whole — speedup = 1 ÷ ((1 − p) + p ÷ s).

Why guessing fails

Every programmer has rebuilt the same wheel: a program is slow, a function looks expensive, so that is the one that gets rewritten — and it turns out the time was somewhere else entirely. This is not a character flaw; it is that human intuition about which line is expensive is bad, because the expensive line is usually doing something invisible: walking a list, rebuilding a string, or waiting on a call.

The whole discipline is one sentence: measure, change one thing, measure again. In Python that means two standard-library tools you already have — cProfile for the whole program, and timeit for one snippet.

Measure the whole program: cProfile

The fastest useful thing you can do is run your script under the profiler and sort by cumulative time:

python -m cProfile -s cumtime your_script.py

You get one row per function, with five columns that matter:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1000    0.412    0.000    6.905    0.007 your_script.py:31(process_record)
  1000000    1.983    0.000    1.983    0.000 your_script.py:58(is_known)
        1    0.004    0.004    6.910    6.910 your_script.py:12(main)

Read it like this:

• tottime — time spent inside this function, not counting the functions it calls. is_known has the biggest tottime: that is where the CPU actually burns.
• cumtime — this function plus everything it called. process_record has the biggest cumtime, because it is the function that calls is_known a million times.
• ncalls — how many times it ran. A tiny function with a huge ncalls is a giant in disguise.

The pattern to look for is the one in that listing: a cheap-looking helper (is_known) called an enormous number of times from inside a loop. That is the shape of most real slowdowns, and it is exactly what a guess would have missed.

The three shapes that cause most slow Python

When you open the profile, look for these three, in this order:

1. A membership test in a loop. if item in my_list: inside a for loop. The in walks the list; the loop walks it again; together they are O(n²). This is the single most common performance bug in beginner-to-intermediate Python.
2. A string grown in a loop. text += chunk inside a loop rebuilds the growing string every pass. ''.join(chunks) builds it once. The += form can be quietly quadratic.
3. A call in a loop that should be out of it. A compile, a file open, a database query, or a regex built inside the loop instead of before it.

Each of these shows up in the profile as a small tottime with a colossal ncalls, which is precisely why a guess misses them: nothing in the code looks heavy, because the cost is in how often it runs.

The classic fix: a list becomes a set

Here is the bug in miniature — the same story the profile above is telling you, stripped down:

# O(n^2): for every row, walk the whole seen list
seen = []
for row in rows:
    if row.key not in seen:      # membership in a list: O(n)
        seen.append(row.key)
        process(row)

# O(n): the set answers "is it there?" in about one step
seen = set()
for row in rows:
    if row.key not in seen:      # membership in a set: O(1)
        seen.add(row.key)
        process(row)

The result is identical, and the difference is not a constant factor — it is the shape of the cost. Growing a list is O(n²); growing a set is O(n). Watch the gap widen as the data grows.

type: line
title: Finding one item, by how many there are (operations)
x: 100, 1000, 10000, 100000
In a list: 100, 1000, 10000, 100000
In a set: 1, 1, 1, 1

Membership in a list: O(n) · walks item by item
Membership in a set: O(1) · about one step, however big the set
Wrapped in a loop: × n · what turns O(n) into O(n squared)
cProfile sort: by cumtime · finds the slow area, not the slow function
timeit: one snippet · run many times, reported precisely

Micro-benchmark the fix: timeit

Before you rewrite anything, time the two spellings against each other on your own data:

python -m timeit -s "data = list(range(100000)); s = set(data)" "99999 in data"
python -m timeit -s "data = list(range(100000)); s = set(data)" "99999 in s"

-s is the setup, run once and not timed; the last argument is the statement, run many times and reported. The list will lose by orders of magnitude at this size, and the ratio is the thing you are shopping for. If the difference is small on your data, keep the clearer code and move on.

The ceiling on any fix: Amdahl’s law

There is a hard limit on what a single optimisation can do, and it is worth knowing before you start, because it stops you spending a day to win two percent.

If a fraction p of the runtime is in the code you are speeding up, and you make that part s times faster, the overall speedup is:

speedup = 1 ÷ ((1 − p) + p ÷ s)

The (1 − p) term is the part you did not touch — it never gets faster, and it sets the ceiling. If only 30% of the time is in your hot function, then even making that function infinitely fast leaves the other 70%:

speedup = 1 ÷ (0.70 + 0) = 1.43×

That is the whole lesson: a 10× faster function is not a 10× faster program unless it was nearly all of the program. The profile tells you where the time is; Amdahl’s law tells you what winning there is actually worth.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <style>
    body { font-family: ui-monospace, Menlo, Consolas, monospace; background: #14130f; color: #e9e4d7; margin: 0; padding: 16px; font-size: 13px; }
    h4 { margin: 0 0 12px; color: #facc15; font-size: 14px; text-transform: uppercase; letter-spacing: .08em; }
    .grid { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; margin-bottom: 12px; }
    label { display: block; color: #b0a899; margin-bottom: 4px; font-size: 11px; text-transform: uppercase; }
    input { width: 100%; box-sizing: border-box; background: #1c1a15; border: 1px solid #332f27; color: #fff; padding: 6px 8px; font-family: inherit; border-radius: 4px; }
    .results { margin-top: 14px; padding: 12px; background: #1c1a15; border: 1px solid #332f27; border-radius: 4px; display: grid; grid-template-columns: repeat(2, 1fr); gap: 10px; }
    .res-box { border-left: 2px solid #5b9cf8; padding-left: 8px; }
    .res-num { font-size: 16px; font-weight: bold; color: #d4756a; }
    .res-lbl { font-size: 10px; color: #7d766a; text-transform: uppercase; }
    .note { margin: 12px 0 0; font-size: 11px; color: #7d766a; line-height: 1.5; }
  </style>
</head>
<body>
  <h4>Amdahl's Law: What Your Fix Is Worth</h4>
  <div class="grid">
    <div><label>Share of time in the slow part (%)</label><input type="number" id="p" value="30" min="1" max="99" oninput="calc()"></div>
    <div><label>How much faster you make it (x)</label><input type="number" id="s" value="10" min="1" max="10000" oninput="calc()"></div>
  </div>
  <div class="results">
    <div class="res-box"><div class="res-num" id="o-speed">&mdash;</div><div class="res-lbl">Overall speedup</div></div>
    <div class="res-box"><div class="res-num" id="o-ceiling">&mdash;</div><div class="res-lbl">Ceiling if the fix were infinite</div></div>
  </div>
  <p class="note" id="note"></p>
  <script>
    function calc() {
      const p = Math.min(99, Math.max(1, parseFloat(document.getElementById('p').value) || 0)) / 100;
      const s = Math.max(1, parseFloat(document.getElementById('s').value) || 1);
      const speed = 1 / ((1 - p) + p / s);
      const ceiling = 1 / (1 - p);
      document.getElementById('o-speed').innerText = speed.toFixed(2) + '\u00d7';
      document.getElementById('o-ceiling').innerText = ceiling.toFixed(2) + '\u00d7';
      document.getElementById('note').innerText = 'A ' + s + '\u00d7 faster function buys a ' + speed.toFixed(2) + '\u00d7 faster program. The other ' + Math.round((1 - p) * 100) + '% never gets faster, and caps you at ' + ceiling.toFixed(2) + '\u00d7.';
      if (window.parent && window.parent.postMessage) {
        window.parent.postMessage({ __orchestra: 'preview', kind: 'height', px: document.body.scrollHeight + 16 }, '*');
      }
    }
    window.addEventListener('load', calc);
  </script>
</body>
</html>

Watch it done

This walkthrough by NeuralNine runs cProfile on a small script and reads the output column by column — which is the part that takes a couple of tries to get comfortable with.

The method, in five lines

The whole approach is deliberately boring, and it is the boringness that makes it work:

1. Profile the whole program and find the function with the largest tottime that you wrote.
2. Check the shape — a membership test in a loop, a string built with +=, or a call that belongs outside the loop.
3. Benchmark the candidate fix with timeit before you touch the code.
4. Estimate the ceiling with Amdahl’s law, so you know what the fix is worth before you spend the afternoon on it.
5. Change one thing, re-run the profile, and keep it only if the number moved.

Profiling is not about being clever. It is about refusing to be clever until you know where the time actually goes.

Frequently asked questions

What is the fastest way to see where a Python program spends its time?

Run it under cProfile and sort by cumulative time: python -m cProfile -s cumtime your_script.py. That prints every function with the time it and everything it called took. Sort by tottime instead to see which function is slow in itself.

What is the difference between tottime and cumtime in cProfile?

tottime is the time spent in the function body itself, excluding the calls it makes. cumtime is that time plus everything the function called. cumtime finds the slow area; tottime finds the slow function.

Why is if x in my_list slow inside a loop?

Testing membership in a list walks the list item by item, so it costs O(n). Inside a loop over n items that becomes O(n²) — doubling the data quadruples the work. A set or dict answers the same question in about one step, O(1).

Should I use timeit or cProfile?

timeit is for a small snippet you want to time on its own, run many times and reported precisely. cProfile is for a whole program, to see the call tree. Use timeit to compare two spellings of the fix, and cProfile to find the fix in the first place.

What is Amdahl’s law?

The rule that bounds any optimisation: if a fraction p of the runtime becomes s times faster, the overall speedup is capped at 1 ÷ ((1 − p) + p ÷ s). If the slow part is only 30% of the time, no change to it — however clever — can speed the program up by more than about 1.4 times.

Is hand-optimising Python worth it?

Only after you have profiled. The slow part is rarely where you expect, and a change made on a hunch usually costs readability for nothing. Profile, change one thing, and keep it only if the number improved.

Where these numbers come from

• cProfile and its columns — Python’s own standard library, documented in the cProfile and pstats pages. The example output above is a real cProfile listing, abbreviated to the three rows that matter.
• The O(n²) and O(n) claim — from how CPython stores the two types: a list is a contiguous array you scan, a set is a hash table you look into. The Python wiki’s time-complexity table states the memberships directly.
• Amdahl’s law — a standard result in performance work, named for Gene Amdahl’s 1967 argument. The formula here is the parallel-speedup form.
• The operation counts in the chart — the shape of the cost, not measured nanoseconds. Which is the point: the shape is what decides whether a fix is worth making, long before you run the benchmark.

This is the same discipline as How to Rename Files with Python and How to Write Regular Expressions: decide what you want precisely, then let a measurement rather than a feeling settle which way of saying it is right.

• Also see: How to Measure the Speed of Light — with a Microwave and a Chocolate Bar
• Also see: How to Read Binary: Bits, Bytes and the Number 255