Python Generator

Source: Dev.to

Generator Functions

Generator functions are a special kind of function that return a lazy iterator.
These objects can be looped over like a list, but unlike lists they do not store their contents in memory.

When a generator function is called, it returns a generator object.
The code inside the function runs only when the generator’s next() (or __next__()) method is invoked, producing values one at a time.

Generator Expressions (Generator Comprehensions)

A generator expression looks almost identical to a list comprehension, but instead of creating a full list in memory it creates a generator object that produces values lazily.

List Comprehension vs. Generator Expression

List comprehension (creates a full list in memory)

squares = [x * x for x in range(5)]
print(squares)
# Output: [0, 1, 4, 9, 16]

All values are computed immediately and stored in memory.

Generator expression (lazy evaluation)

squares = (x * x for x in range(5))
print(squares)          # <generator object at 0x...>

Nothing is computed yet.
Values are generated only when you iterate over the generator.

How to use a generator expression?

You must iterate over it, e.g.:

for num in squares:
    print(num)

Or convert it to a list (which forces evaluation):

print(list(squares))

Memory‑Usage Example (Important)

import sys

lst = [x for x in range(1_000_000)]
gen = (x for x in range(1_000_000))

print(sys.getsizeof(lst))   # large
print(sys.getsizeof(gen))   # small

The generator uses much less memory because it does not store all elements at once.

“Without Calling a Function”

Normally you would create a generator with a function:

def my_generator():
    for x in range(5):
        yield x * x

With a generator expression you can skip the function definition:

gen = (x * x for x in range(5))

Very Common Use Case – Passing Directly into Functions

Many built‑in functions accept any iterable, so you can feed a generator expression straight into them:

total = sum(x * x for x in range(1_000_000))

• No extra brackets needed
• Memory‑efficient
• Clean syntax

The yield Statement

yield is similar to return, but instead of terminating the function it pauses it, saving its state and returning a value to the caller. When the generator is resumed (via next() or a for loop), execution continues right after the yield.

Key points

• The generator’s local variables, instruction pointer, and internal stack are saved.
• Multiple yield statements can be used to produce a sequence of values.
• return ends the generator completely, raising StopIteration.

When to Use Generator Expressions?

• ✅ Large datasets
• ✅ Streaming data
• ✅ Single‑pass iteration
• ✅ Memory‑sensitive applications

Avoid them when you need:

• ❌ Random indexing
• ❌ Multiple passes over the data

How Lazy Evaluation Works Internally

Lazy evaluation means values are computed only when needed. In Python this is achieved through iterators and generators.

Eager vs. Lazy (mental model)

Eager evaluation

data = [x * 2 for x in range(5)]

• The loop runs immediately, all values are computed, and the list is stored in memory.

Lazy evaluation

data = (x * 2 for x in range(5))

• Nothing is computed yet; only a generator object is created.
• Values are produced one‑by‑one when iterated.

What a Generator Really Is

A generator is essentially a state machine:

1. Pauses execution at each yield.
2. Saves the current instruction pointer and local variables.
3. Returns the yielded value.
4. Resumes later from the saved point.

Step‑by‑step execution example

def squares():
    for i in range(3):
        yield i * i

g = squares()   # generator created, no code executed yet
next(g)          # → 0
next(g)          # → 1
next(g)          # → 4
next(g)          # raises StopIteration

Each next(g) resumes execution until the next yield, then pauses again.

Why Generators Are Memory‑Efficient

• range(1_000_000) stores only start, stop, and step.
• (x * x for x in range(1_000_000)) stores a reference to the range, the current index, and the execution state.

Thus memory usage stays constant, regardless of the number of items produced.

Lazy Evaluation in Built‑ins

Example

m = map(lambda x: x * x, range(10))
# No computation occurs until `m` is iterated.

How StopIteration Ends Lazy Evaluation

When a generator finishes, Python raises StopIteration. The iteration protocol (e.g., a for loop) catches this exception and stops the loop.

gen = (x for x in range(3))
list(gen)   # [0, 1, 2]
list(gen)   # []  (generator is exhausted)

A generator is single‑pass; once it reaches the end, it cannot be rewound unless you create a new generator.

Python Generators – Advanced Control Methods

Generators let you produce values lazily, pausing execution at each yield. Beyond simple iteration, Python provides three control methods that let you interact with a generator from the outside:

1. .send(value) – Send data into a generator

Normally a generator only yields values outward.
send() injects a value back into the generator, which is returned by the most recent yield expression.

def counter():
    value = yield 0          # first yield
    while True:
        value = yield value + 1

gen = counter()

print(next(gen))       # start generator → yields 0
print(gen.send(10))    # sends 10 → yields 11
print(gen.send(20))    # sends 20 → yields 21

Key rules

• The first call must be next(gen) or gen.send(None).
• send(x) assigns x to the last yield expression.

2. .throw(exception) – Raise an exception inside the generator

throw() injects an exception at the point where the generator is paused.
If the generator catches the exception, it can handle it; otherwise the exception propagates outward.

def generator():
    try:
        while True:
            yield "running"
    except ValueError:
        yield "ValueError handled"

gen = generator()
print(next(gen))                     # → running
print(gen.throw(ValueError))         # → ValueError handled

Typical use‑cases

• Cancel ongoing work
• Signal error conditions
• Interrupt long‑running generators

3. .close() – Stop the generator gracefully

Calling close() raises a GeneratorExit inside the generator, allowing it to perform any necessary cleanup in a finally block.

def generator():
    try:
        while True:
            yield "working"
    finally:
        print("Cleaning up resources")

gen = generator()
print(next(gen))   # → working
gen.close()        # triggers cleanup

Output

working
Cleaning up resources

Lifecycle Summary

To Summarize

A generator in Python is a function that:

• Uses the yield keyword.
• Produces values one at a time.
• Remembers its local state between executions.

When Python encounters yield it:

1. Returns the yielded value to the caller.
2. Pauses execution, saving the current local state.
3. Resumes from that exact point on the next iteration (or when send, throw, or close is invoked).