What is the common pitfall with Attempting to Re-Use an Exhausted Generator?

Generators are single-use streams. Once exhausted (StopIteration raised), they remain empty. Exam trap note: Exams test why passing a generator to two consecutive loops produces empty results on the second loop.

What are the core concepts of Python Iterators & Generators?

An iterable is an object with an `__iter__()` method that returns an iterator. An iterator is an object with a `__next__()` method that returns the next value. When no elements remain, `__next__()` raises the `StopIteration` exception, which Python for loops catch automatically.

← Previous: Python Inheritance & Polymorphism All Modules Next: Python Scope & LEGB Rule →

MOD-20 · Sequence #20 · Advanced Python

Python Iterators & Generators

Iterators implement the __iter__ and __next__ protocol to produce items sequentially. Generators use the yield keyword to produce values lazily on demand, pausing stack execution and offering O(1) memory efficiency for processing massive data streams.

Live Interactive Code Snippet (Python 3.12 CPython WASM) Open in Practice Mode →

def countdown(n):
    while n > 0:
        yield n
        n -= 1

print(list(countdown(3)))  # [3, 2, 1]

Core Principles & Mental Models

1. The Iterator Protocol (iter & next)

StopIteration Exception Mechanics

An iterable is an object with an `__iter__()` method that returns an iterator.
An iterator is an object with a `__next__()` method that returns the next value.
When no elements remain, `__next__()` raises the `StopIteration` exception, which Python for loops catch automatically.

numbers = [10, 20]
it = iter(numbers)
print(next(it))  # 10
print(next(it))  # 20
# next(it) -> Raises StopIteration!

💡 Key Takeaway: For loops in Python are syntactic sugar over calling iter() and repeatedly calling next() until StopIteration.

2. Generators and the `yield` Keyword

Coroutine State Suspension and Resumption

Any function containing the `yield` keyword is a generator function.
Calling a generator function does not execute its code; it returns a generator object.
Each call to `next()` executes code until hitting `yield <value>`, yielding the value and freezing execution state (local variables, instruction pointer) until the next request.

def fibonacci_stream(limit):
    a, b = 0, 1
    count = 0
    while count < limit:
        yield a
        a, b = b, a + b
        count += 1

for num in fibonacci_stream(6):
    print(num, end=" ")
print()

💡 Key Takeaway: Generators turn memory-bound O(N) pipelines into constant O(1) memory streaming pipelines.

Interactive Walkthrough Examples

Generator Expressions vs. List Comprehensions (Intermediate)

Comparing memory consumption of generator expressions `(x for x in ...)` vs list comprehensions.

import sys
# List comprehension: builds all 100,000 items in memory
list_comp = [x ** 2 for x in range(100_000)]
# Generator expression: creates values on demand
gen_exp = (x ** 2 for x in range(100_000))

print("List memory:", sys.getsizeof(list_comp), "bytes")
print("Generator memory:", sys.getsizeof(gen_exp), "bytes")
print("First 3 from generator:", [next(gen_exp), next(gen_exp), next(gen_exp)])

Explanation: Generator expressions use parentheses and compute values lazily, requiring constant memory.

Common Misconceptions & Exam Traps

Common Pitfall: Attempting to Re-Use an Exhausted Generator

Generators are single-use streams. Once exhausted (StopIteration raised), they remain empty.

Buggy Code

gen = (x for x in [1, 2, 3])
print("Sum:", sum(gen))       # 6
print("Max:", max(gen))       # ValueError: max() arg is an empty sequence!

Correct Solution

data = [1, 2, 3]
print("Sum:", sum(data))      # 6
print("Max:", max(data))      # 3 (Re-iterable collection)

Exam Trap Note: Exams test why passing a generator to two consecutive loops produces empty results on the second loop.

Conceptual Knowledge Self-Check Quiz

Q1: What exception signals to a Python for loop that an iterator has reached the end of its elements?

IndexError
StopIteration
GeneratorExit
EOFError

The iterator protocol relies on raising `StopIteration` when no further elements can be yielded.

Algorithmic Practice Challenges

Idiomatic Best Practices

✅ Use generator functions for reading large files line-by-line.
✅ Use generator expressions `(expr for x in seq)` when passing data directly to aggregators like `sum()`, `any()`, `all()`.

← Python Inheritance & Polymorphism Take Assessed Test → Python Scope & LEGB Rule →

All 26 Curriculum Modules

MOD-01: Python Syntax & Indentation — Whitespace blocks, execution statements, and semicolon absence in Python.
MOD-02: Python Comments — Single-line comments (#) and multiline docstrings.
MOD-03: Python Variables & Dynamic Binding — Variable creation, dynamic typing, unpacking, and multiple assignment.
MOD-04: Python Data Types — Built-in fundamental types: int, float, str, bool, list, tuple, dict, set, None.
MOD-05: Python Numbers & Type Casting — Int, float, complex numbers, and casting via int(), float(), str().
MOD-06: Python Strings & Slicing — String arrays, indexing [start:stop:step], negative indices, and string methods.
MOD-07: Python Booleans & Logical Evaluation — Boolean truth values, truthy vs falsy values, and logical operators.
MOD-08: Python Operators — Arithmetic, assignment, comparison, logical, identity (is), and membership (in).
MOD-09: Python Lists (Arrays) — Ordered, mutable sequences, indexing, append, insert, remove, pop, and list comprehensions.
MOD-10: Python Tuples — Ordered, immutable sequences, unpacking, and tuple hashing for dictionary keys.
MOD-11: Python Sets — Unordered, unindexed collections of unique elements; union, intersection, difference.
MOD-12: Python Dictionaries — Key-value associative mappings, keys(), values(), items(), and dict comprehensions.
MOD-13: Python If...Else & Branching — Conditional branching, elif ladder, nested conditions, and ternary conditional operators.
MOD-14: Python While Loops — Conditional loops, break, continue, and the while...else loop construct.
MOD-15: Python For Loops & Range — Sequence iteration, range(start, stop, step), enumerate(), and zip().
MOD-16: Python Functions & Parameters — def declarations, positional args, keyword args, *args, **kwargs, and return statements.
MOD-17: Python Lambda Functions — Anonymous inline functions, map(), filter(), and sorted(key=lambda).
MOD-18: Python Classes & Objects (OOP) — Class definition, the __init__ constructor, the self receiver, and instance methods.
MOD-19: Python Inheritance & Polymorphism — Subclassing, super() initializer, method overriding, and polymorphism.
MOD-20: Python Iterators & Generators — __iter__ and __next__ protocols, and generator functions with the yield keyword.
MOD-21: Python Scope & LEGB Rule — Local, Enclosing, Global, and Built-in namespaces; global and nonlocal keywords.
MOD-22: Python Modules & Packages — import, from ... import ..., as alias, dir() inspection, and __name__ == "__main__".
MOD-23: Python Dates & Math — datetime module (now, strftime, strptime) and math functions (pow, sqrt, ceil, floor).
MOD-24: Python JSON & Regular Expressions — json.loads(), json.dumps(), and the re module (search, findall, sub, split).
MOD-25: Python Try...Except & Error Handling — Exception catching with try, except, else, finally, and raising custom exceptions.
MOD-26: Python File Handling — File open modes ("r", "w", "a", "x"), with context manager, and os removal.