🚀AdvancedAdvanced Patterns
Memory Management & Performance
__slots__, generators for memory efficiency, cProfile, timeit, and practical optimisation techniques.
Overview
Understanding Python's memory model and performance characteristics lets you write code that scales. Python uses reference counting for memory management (with a cycle collector for circular references). Every Python object carries header overhead even a simple integer is 28 bytes.
**Memory optimisation**: `__slots__` replaces the per-instance `__dict__` with a fixed-size structure, dramatically reducing memory for classes with many instances. Generators produce values lazily instead of materialising entire lists. `array.array` and `numpy` arrays store homogeneous data without per-element Python overhead.
**Performance profiling**: always measure before optimising. `timeit` benchmarks small snippets. `cProfile` profiles entire programs and shows where time is spent. `line_profiler` (third-party) shows per-line timing. `tracemalloc` tracks memory allocations.
Common bottlenecks: attribute lookup, function call overhead, string concatenation in loops (`+` vs `join`), list vs generator, and I/O. Know your bottleneck before reaching for multiprocessing.
Key Points
- •`__slots__ = ("x", "y")` eliminates `__dict__` per instance saves ~50-200 bytes per object
- •`sys.getsizeof(obj)` returns object size in bytes (shallow); `tracemalloc` tracks allocations
- •Generator expressions vs list comprehensions: `(x for x in it)` uses O(1) memory vs O(n)
- •`timeit.timeit("expr", number=1000000)` accurate micro-benchmarks
- •`cProfile.run("fn()")` or `python -m cProfile script.py` call-level profiling
- •String building: `"".join(parts)` is O(n); repeated `s += ...` is O(n²)
- •`array.array("d", [1.0, 2.0])` compact, type-homogeneous numeric array
- •`functools.lru_cache` memoise pure functions; eliminates redundant computation
- •`locals()` lookup is faster than `globals()` assign frequently used globals to locals inside hot loops
Examples
__slots__ memory reduction for many instances
import sys
class WithDict:
def __init__(self, x, y):
self.x = x
self.y = y
class WithSlots:
__slots__ = ("x", "y")
def __init__(self, x, y):
self.x = x
self.y = y
d = WithDict(1, 2)
s = WithSlots(1, 2)
print(f"With __dict__: {sys.getsizeof(d) + sys.getsizeof(d.__dict__)} bytes")
print(f"With __slots__: {sys.getsizeof(s)} bytes")
# __slots__ is typically 120-160 bytes vs 240-280 bytesGenerator vs list memory for large sequences
import sys
# List materialises all values in memory
big_list = [x * x for x in range(1_000_000)]
print(f"List size: {sys.getsizeof(big_list):,} bytes")
# Generator produces one value at a time
big_gen = (x * x for x in range(1_000_000))
print(f"Generator size: {sys.getsizeof(big_gen)} bytes")
# Both produce the same result
import itertools
first_five_list = big_list[:5]
first_five_gen = list(itertools.islice(big_gen, 5))
print(first_five_list)
print(first_five_gen)Profiling with timeit and cProfile
import timeit
# Benchmark string join vs concatenation
def join_method(n):
parts = []
for i in range(n):
parts.append(str(i))
return "".join(parts)
def concat_method(n):
s = ""
for i in range(n):
s += str(i)
return s
n = 1000
join_time = timeit.timeit(lambda: join_method(n), number=1000)
concat_time = timeit.timeit(lambda: concat_method(n), number=1000)
print(f"join: {join_time:.4f}s")
print(f"concat: {concat_time:.4f}s")
print(f"join is {concat_time/join_time:.1f}x faster")Common Mistakes
- Optimising before profiling guessing the bottleneck is almost always wrong; measure first
- Using `__slots__` on classes that need dynamic attribute assignment or multiple inheritance (slots restrict flexibility)
- Calling `list(generator)` immediately after creating it defeats the memory benefit of the generator
Note
The 80/20 rule of performance: 80% of execution time is in 20% of the code. Profile with `cProfile`, find the hotspot, then optimise only that. Premature optimisation wastes time and creates brittle code.
Exercises
Write Python code to solve each exercise. Run it in your browser and check your output.
Loading editor…
Completed: 0 / 5 exercises
Knowledge Check
Question 1 of 4