Generators in Python

One of the first things you learn in python is iterating over numbers using range. Learn about how it works, and how to write your own generators to reduce memory footprint of your code. 🧵

Python does not have a foor loop, it has a for-each loop. You can iterate over data structures like lists or dictionaries, but you can also iterate over things that aren’t a datastructure in memory and that’s where generators come in.

Example 0: Iterating using range

for i in range(0, 10):
    print(i)

Generators like range only produce one item at a time. They know where they are in a sequence, and how to produce next item. This makes them useful for tasks such as reading large files, where you only need to process one record at a time.

Now, let’s create an alternative implementation of range. To define a generator, use the yield keyword in a function definition. When we iterate, it will yield items one by one until it reaches return, at which point it becomes exausthed.

Example 1: Alternative implementation of range

from typing import Generator


def my_range(start: int, end: int) -> Generator[int, None, None]:
    current = start
    while current < end:
        yield current
        current += 1
    return None


for i in my_range(0, 10):
    print(i)

Exhausted generator means that the iteration stopped. Once the generator has produced all of its values, it can no longer be used. This means you can’t go back and revisit previous values. Generators are one-time use objects.

Another benefit is that you don’t always have to exhaust a generator, and only process what you need to. For example, you might search for a line in a large file and stop iterating when you find it.

Here are a two additional examples of generators:

• fibonnachi sequence using generators
• reading a potentionally large file line-by-line, because it might not fit in memory - This is actually built into file object in Python!

Example 2: Fibonacci in python using a generator

from typing import Generator


def fibonacci_numbers(n) -> Generator[int, None, None]:
    """Return first n fibonacci numbers as a generator."""
    a, b = 0, 1
    for _ in range(0, n):
        yield a
        a, b = b, a + b


for i in fibonacci_numbers(10):
    print(i)

Example 3: Reading file line by line using generators

File object in python can be iterated trough, because it acts a generator of lines from the file! The following is a demonstration where we look for a specific line in a file.

def find_line_in_file(filename: str, match: str) -> bool:
    """Match a line in file."""
    f = open(filename, "r")

    for line in f:
        if line.strip("\n") == match:
            print("found it!")
            f.close()
            return True

    f.close()
    return False


found = find_line_in_file(filename="longfile.txt", match="ten")
print(found)

check the code examples on GitHub