âš¡IntermediateFunctional Python
Generators and Iterators
Memory-efficient sequences with yield, generator expressions, and the iterator protocol.
Overview
An **iterator** is any object that implements `__iter__()` and `__next__()`. When Python executes a for loop, it calls `iter(obj)` to get an iterator, then repeatedly calls `next()` on it until `StopIteration` is raised.
A **generator** is a special kind of iterator defined using `yield`. When a function contains `yield`, calling it returns a generator object instead of running the body immediately. Each call to `next()` resumes execution from where it last yielded, pausing again at the next `yield`. The function's local state is preserved between yields.
Generators are lazy they produce values one at a time, only when asked. This makes them memory-efficient for large sequences: a generator that yields 1 million numbers uses almost no memory, whereas a list of 1 million numbers uses ~8 MB.
The `yield from` syntax delegates to another iterable, and generator expressions `(x**2 for x in range(n))` create inline generators without a full function definition.
Key Points
- •A generator function contains `yield` calling it returns a generator, not the result
- •Execution pauses at each `yield` and resumes on the next `next()` call
- •Generator state (local variables) is preserved between yields
- •`yield from iterable` delegates to another generator/iterable cleanly
- •Generator expression: `(expr for x in iterable)` like a list comprehension but lazy
- •`next(gen, default)` get next value or return default instead of raising StopIteration
- •Use generators for infinite sequences, large data streams, and pipelines
Examples
Basic generator function
def countdown(n):
print("Starting countdown")
while n > 0:
yield n # pause and return n
n -= 1
print("Done!")
gen = countdown(3)
print(next(gen)) # Starting countdown
3
print(next(gen)) # 2
print(next(gen)) # 3
# print(next(gen)) # would raise StopIteration after "Done!"
# for loop handles StopIteration automatically
for value in countdown(3):
print(value, end=" ") # 3 2 1Infinite generator and generator expressions
def fibonacci():
a, b = 0, 1
while True: # infinite no StopIteration
yield a
a, b = b, a + b
fib = fibonacci()
first_ten = [next(fib) for _ in range(10)]
print(first_ten) # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
# Generator expression lazy, no list created
squares_gen = (x**2 for x in range(1_000_000))
print(next(squares_gen)) # 0
print(next(squares_gen)) # 1
# Only 2 values computed so far!yield from and pipeline composition
def flatten(nested):
for item in nested:
if isinstance(item, list):
yield from flatten(item) # recursively yield from sub-lists
else:
yield item
data = [1, [2, 3], [4, [5, 6]], 7]
print(list(flatten(data))) # [1, 2, 3, 4, 5, 6, 7]
# Generator pipeline each stage is lazy
def read_numbers():
yield from range(1, 11)
def only_evens(nums):
return (n for n in nums if n % 2 == 0)
def squared(nums):
return (n**2 for n in nums)
pipeline = squared(only_evens(read_numbers()))
print(list(pipeline)) # [4, 16, 36, 64, 100]Common Mistakes
- Trying to index a generator: gen[0] raises TypeError generators are not subscriptable
- Exhausting a generator and expecting it to reset once a generator is done, it's done. Re-call the generator function for a new one
- Forgetting that a generator function returns a generator object the body doesn't run until you call next()
Note
When you have a function that builds and returns a large list, consider making it a generator instead just replace `result.append(x)` with `yield x`. Callers using `for item in fn()` work identically, but memory usage drops to near-zero.
Exercises
Write Python code to solve each exercise. Run it in your browser and check your output.
Loading editor…
Completed: 0 / 5 exercises
Knowledge Check
Question 1 of 4