Chapter 11: Iterators and Generators

Iterators and generators are advanced Python features that allow for efficient looping and data generation. They enable you to handle large datasets and compute values lazily, improving performance and memory usage.

Iterators

An iterator is an object that implements the __iter__() and __next__() methods. It represents a sequence of values, returning one value at a time when iterated.

Example of an Iterator:

# Create an iterator
numbers = iter([1, 2, 3, 4, 5])

# Access elements using next()
print(next(numbers))  # Output: 1
print(next(numbers))  # Output: 2

# Iterate over the rest using a loop
for num in numbers:
    print(num)  # Output: 3, 4, 5

Key Methods:

Method	Description
__iter__()	Returns the iterator object itself.
__next__()	Returns the next item in the sequence or raises StopIteration.

Custom Iterators

You can create custom iterators by defining a class that implements the __iter__() and __next__() methods.

Example:

class Counter:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.end:
            raise StopIteration
        self.current += 1
        return self.current - 1

# Using the custom iterator
counter = Counter(1, 5)
for num in counter:
    print(num)  # Output: 1, 2, 3, 4, 5

Generators

A generator is a special type of iterator created using functions and the yield keyword. Generators are used to lazily produce values one at a time, as needed.

Creating a Generator:

def generate_numbers(n):
    for i in range(1, n + 1):
        yield i

# Using the generator
for num in generate_numbers(5):
    print(num)  # Output: 1, 2, 3, 4, 5

Key Differences Between Generators and Iterators:

1. Generators are defined using functions and the yield keyword.

2. Generators automatically implement the __iter__() and __next__() methods.

Generator Expressions

Generator expressions are similar to list comprehensions but produce values lazily.

Example:

# Generator expression
squares = (x ** 2 for x in range(1, 6))

# Iterate over the generator
for square in squares:
    print(square)  # Output: 1, 4, 9, 16, 25

Advantages of Generators

1. Memory Efficiency: Generate values on the fly instead of storing them in memory.

2. Lazy Evaluation: Values are computed only when needed.

3. Readable Code: Easier to implement compared to custom iterators.

Practical Use Cases

1. Reading Large Files: Process a file line by line without loading the entire file into memory.

   def read_large_file(filename):
       with open(filename, "r") as file:
           for line in file:
               yield line.strip()
   
   for line in read_large_file("example.txt"):
       print(line)

2. Infinite Sequences: Generate an unbounded sequence of values.

   def infinite_numbers(start):
       while True:
           yield start
           start += 1
   
   for num in infinite_numbers(1):
       if num > 5:
           break
       print(num)  # Output: 1, 2, 3, 4, 5

3. Efficient Data Pipelines: Process data streams incrementally.

Exercises

Exercise 1:

Write a custom iterator that generates the Fibonacci sequence up to a given number n.

Solution:

class Fibonacci:
    def __init__(self, n):
        self.n = n
        self.a, self.b = 0, 1

    def __iter__(self):
        return self

    def __next__(self):
        if self.a > self.n:
            raise StopIteration
        result = self.a
        self.a, self.b = self.b, self.a + self.b
        return result

# Using the iterator
for num in Fibonacci(10):
    print(num)  # Output: 0, 1, 1, 2, 3, 5, 8

Exercise 2:

Write a generator function to yield even numbers up to a given number n.

Solution:

def even_numbers(n):
    for i in range(2, n + 1, 2):
        yield i

# Using the generator
for num in even_numbers(10):
    print(num)  # Output: 2, 4, 6, 8, 10

Exercise 3:

Use a generator expression to create a sequence of cubes for numbers 1 through 5.

Solution:

cubes = (x ** 3 for x in range(1, 6))
for cube in cubes:
    print(cube)  # Output: 1, 8, 27, 64, 125

Best Practices

1. Use generators for processing large datasets or streams to save memory.

2. Combine generator expressions with functions like sum() and max() for concise calculations.

3. Prefer yield over returning large lists when possible.

4. Test custom iterators to handle edge cases, such as empty sequences.

In the next chapter, we will explore decorators and context managers, which are powerful tools for enhancing and managing Python functions and resources.