Python Generators and the yield Keyword, Made Simple

Quick Answer

A Python generator is a function that uses yield instead of return, so it produces values one at a time instead of building a whole list in memory. Each time you loop over it, the function runs up to the next yield, hands back that value, and pauses. Because only one item exists at a time, generators use almost no memory even for millions of items or huge files. Use them for large or streaming data, and use a normal list when you need indexing or multiple passes.

What is a Python generator?

If you have written a Python function before, you know it usually does its work and hands back an answer with return. Python generators work differently. A generator is a special kind of function that produces a sequence of values one at a time, pausing after each one and remembering exactly where it left off.

The word that makes this happen is yield. The moment you put a yield anywhere inside a function, Python stops treating it as an ordinary function and turns it into a generator. Instead of computing everything up front and returning a big list, it becomes lazy: it only does work when you ask for the next value.

This one idea solves a very practical problem. When you deal with large data, such as a file with millions of lines, building the whole thing in memory can slow your program to a crawl or crash it. Generators let you walk through the data piece by piece using almost no memory.

yield vs return: two ways to hand back data

Let's compare the two styles directly. Here is a function that builds and returns a list of square numbers:

def squares_list(n):
    result = []
    for i in range(n):
        result.append(i * i)
    return result

print(squares_list(5))   # [0, 1, 4, 9, 16]

And here is the generator version. Notice it has no list and no append — just yield:

def squares_gen(n):
    for i in range(n):
        yield i * i

print(list(squares_gen(5)))   # [0, 1, 4, 9, 16]

Both give the same numbers, but they behave very differently. return runs the whole loop, builds the entire list, and hands it back in one go — then the function is finished. yield hands back a single value and pauses the function with all its variables intact, ready to continue from that exact spot the next time a value is requested.

How yield pauses and resumes a function

To really see the pause-and-resume behaviour, call the generator function and step through it by hand with the built-in next():

gen = squares_gen(3)

print(next(gen))   # 0
print(next(gen))   # 1
print(next(gen))   # 4
print(next(gen))   # raises StopIteration

Calling squares_gen(3) does not run any of your loop yet. It returns a generator object. Each next() resumes the function, runs until it hits the next yield, and freezes again. When the loop finishes and there is nothing left to yield, Python raises StopIteration.

You almost never call next() by hand, though. A normal for loop does it for you and stops cleanly at the end:

for value in squares_gen(3):
    print(value)   # prints 0, then 1, then 4

Why generators save memory on large data

This is the reason generators matter. A list stores every element at the same time, so a list of one million numbers holds one million numbers in memory. A generator holds just one value at a time plus a little bookkeeping, no matter how many values it will eventually produce.

import sys

nums_list = [i for i in range(1_000_000)]
nums_gen  = (i for i in range(1_000_000))

print(sys.getsizeof(nums_list))   # about 8,000,000+ bytes (~8 MB)
print(sys.getsizeof(nums_gen))    # about 200 bytes

The list takes roughly eight megabytes. The generator takes a couple of hundred bytes — and that number stays flat whether you generate a thousand items or a billion. You trade a little extra work per item for a huge saving in memory, which is almost always a good deal when the data is large.

Generator expressions: the quick version

You do not always need a full function. For simple cases Python gives you a generator expression, which looks exactly like a list comprehension but with round brackets () instead of square brackets [].

# List comprehension - builds the whole list now
squares = [x * x for x in range(10)]

# Generator expression - produces values lazily
squares = (x * x for x in range(10))

The best part is that many built-in functions accept a generator directly, so you can process huge sequences without ever creating a list in between:

# No million-item list is ever built
total = sum(x * x for x in range(1_000_000))
print(total)

When a generator expression is the only argument to a function, you can even drop the extra brackets, as with sum(...) above. It reads cleanly and stays memory-friendly.

Streaming a large file, line by line

Here is where generators shine in real projects. Suppose you have a log file that is several gigabytes in size and you want only the lines that contain the word ERROR. Reading the whole file into a list would be a disaster. A generator reads it one line at a time.

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")

def error_lines(path):
    for line in read_lines(path):
        if "ERROR" in line:
            yield line

for line in error_lines("app.log"):
    print(line)

No matter how big app.log is — one megabyte or fifty gigabytes — this code uses about the same tiny amount of memory, because only one line exists at a time. Notice how error_lines pulls from read_lines: generators chain together neatly into a pipeline, each stage passing values to the next without storing everything.

A file object is actually already a lazy iterator, so for line in f reads one line at a time on its own. Wrapping it in your own generator just lets you add filtering, cleaning, or parsing steps in a readable way.

Common gotchas to watch out for

Generators are powerful, but a few surprises catch beginners. Keep these in mind.

They run only once. After you loop through a generator, it is exhausted. Looping again gives you nothing:

gen = squares_gen(3)
print(list(gen))   # [0, 1, 4]
print(list(gen))   # [] - already used up

If you need the values more than once, either recreate the generator by calling the function again, or store the results in a list with list(gen).

No indexing or len(). You cannot write gen[2] or len(gen), because the values do not exist until you ask for them. If you need random access or a count, a list is the right tool.

Nothing runs until you iterate. Because generators are lazy, the code inside them (including print statements or errors) may not run when you expect. The work happens only as you consume values.

When to use a generator vs a list

So which should you reach for? Here is a simple guide.

The rule of thumb: use a generator when data is large, streamed, or infinite, or when you are building a step-by-step processing pipeline. Use a plain list when the data is small and you need to index into it, check its length, or loop over it several times.

Generators are one of those Python features that feel small but change how you write code once they click. If you want to practise them with guided exercises and build real projects, our free Python course walks you through generators, iterators, and file handling step by step.

Frequently Asked Questions