Iterators and Generators: Handling Data One Item at a Time

The Token Machine at the Hospital

Picture the outpatient department of a busy government hospital. Hundreds of patients arrive every morning, and each one needs a token number. Imagine the hospital printed one million tokens at 7 AM, just in case. It would waste paper, fill a room and take hours. Instead, there is a small machine at the counter. When a patient presses the button, it prints the next number, and only then. The machine only remembers one thing: the last number it gave.

Python has the same idea. A list is like the million printed tokens: every item is made and stored in memory at once. An iterator is like the token machine: it gives you the next item only when you ask. A generator is the easiest way to build your own token machine.

This matters in real programs. Log files, sales reports and sensor readings can have millions of lines. If you load all of them into a list, your program can become slow or even crash. If you handle them one item at a time, a small laptop can process a huge file.

Iterables and Iterators

Two words sound alike but mean different things:

The built-in function iter() asks an iterable for an iterator. The built-in function next() asks the iterator for one item. When nothing is left, the iterator raises a special exception called StopIteration. It is not a bug: it is simply the signal for “finished”.

departments = ["OPD", "Dental", "Eye"]

counter = iter(departments)   # ask the list for an iterator

print(next(counter))   # OPD
print(next(counter))   # Dental
print(next(counter))   # Eye

try:
    next(counter)          # nothing is left
except StopIteration:
    print("No more items")  # No more items

What a for loop does behind the scenes

You have written many for loops. Now you can see how they work. Python calls iter() on your iterable once, then calls next() again and again, and quietly stops when it sees StopIteration. These two loops do exactly the same job:

patients = ["Ravi", "Meena", "Arjun"]

# What a for loop really does, step by step
it = iter(patients)
while True:
    try:
        name = next(it)
    except StopIteration:
        break
    print("Calling", name)

# Calling Ravi
# Calling Meena
# Calling Arjun

# The same thing, the normal way
for name in patients:
    print("Calling", name)

This is why a for loop works on so many different things. Lists, strings, files and dictionaries all follow the same simple rule: give me an iterator, and the iterator gives me items.

Writing a Generator with yield

A generator function looks like a normal function, but it uses the keyword yield instead of (or as well as) return. Calling it does not run the body. It gives you a generator object, which is an iterator. Each time you call next() on it, the body runs until the next yield, hands out that value, and then pauses. All its local variables stay as they were. The next call continues from exactly that point.

Here is our hospital token machine. Watch the order of the printed lines carefully:

def token_counter(start=1):
    number = start
    while True:
        print(f"  (machine prints token {number})")
        yield number
        number += 1

machine = token_counter()
print(type(machine))   # <class 'generator'>

print("Patient 1 gets", next(machine))
print("Patient 2 gets", next(machine))
print("Lunch break...")
print("Patient 3 gets", next(machine))

#   (machine prints token 1)
# Patient 1 gets 1
#   (machine prints token 2)
# Patient 2 gets 2
# Lunch break...
#   (machine prints token 3)
# Patient 3 gets 3

Notice three things. First, while True looks like an endless loop, but it is safe, because the function pauses at every yield. Second, the machine prints a token only when a patient asks. Third, the generator remembers number between calls, even during the lunch break.

Generators that finish

Most generators do end. When the function body reaches its end (or a return), Python raises StopIteration for you, and any for loop stops neatly. You can pass a finite generator to any function that loops, such as list(), sum() or max().

def todays_tokens(prefix, how_many):
    for number in range(1, how_many + 1):
        yield f"{prefix}-{number:03}"

for token in todays_tokens("EYE", 4):
    print(token)

# EYE-001
# EYE-002
# EYE-003
# EYE-004

# A generator can feed any function that loops
print(list(todays_tokens("ENT", 3)))   # ['ENT-001', 'ENT-002', 'ENT-003']

Generator Expressions and Memory

You already know list comprehensions from the Comprehensions lesson. Replace the square brackets with round brackets and you get a generator expression. It describes the same values, but it produces them one by one instead of building a full list.

The function sys.getsizeof() tells you how many bytes an object takes. Compare a list of one million squares with a generator of the same squares:

import sys

squares_list = [n * n for n in range(1_000_000)]   # square brackets: a list
squares_gen = (n * n for n in range(1_000_000))    # round brackets: a generator

print("List size:", sys.getsizeof(squares_list), "bytes")
print("Generator size:", sys.getsizeof(squares_gen), "bytes")
# List size: about 8 million bytes
# Generator size: about 100 to 200 bytes

# Both give the same total
print(sum(squares_list))   # 333332833333500000
print(sum(squares_gen))    # 333332833333500000

# A generator expression can go straight into a function call
print(max(len(word) for word in ["fever", "cough", "headache"]))   # 8

The list holds a million numbers, so it needs millions of bytes. The generator only holds its current position and the rule for making the next value, so it stays tiny, no matter how many items it will produce. The exact numbers depend on your Python version, but the difference is always huge.

When should you use a list instead? Use a list when you need to loop over the data more than once, use an index like items[5], sort it, or check its length. Use a generator when you go through the data once from start to end, especially when the data is large.

Reading a Big File Line by Line

A file object is already an iterator of lines. When you write for line in f, Python reads one line, gives it to you, and forgets it before reading the next. This is the right way to handle large files. By contrast, f.readlines() and f.read() load the whole file into memory at once.

The playground starts in an empty folder, so this example first creates a sample visit log, then reads it back one line at a time:

import random

# Step 1: create a sample visit log with 50,000 lines (a real one could be much bigger)
departments = ["OPD", "Dental", "Eye", "ENT", "Skin"]
with open("visits.txt", "w") as f:
    for number in range(1, 50_001):
        f.write(f"{number},{random.choice(departments)}\n")

# Step 2: read it one line at a time. Only one line is in memory at once.
eye_visits = 0
with open("visits.txt") as f:
    for line in f:              # a file object is an iterator of lines
        token, dept = line.strip().split(",")
        if dept == "Eye":
            eye_visits += 1

print("Eye department visits:", eye_visits)   # about 10000 (it is random)

# The same count with a generator expression
with open("visits.txt") as f:
    dental = sum(1 for line in f if line.strip().endswith(",Dental"))
print("Dental visits:", dental)               # about 10000

itertools: Ready-Made Iterator Tools

The standard library module itertools has many small tools that work with iterators. Four are useful from day one:

from itertools import count, islice, cycle, chain

# count(): numbers that never end, starting where you choose
tokens = count(101)
print(next(tokens), next(tokens), next(tokens))   # 101 102 103

# islice(): take only part of any iterator, even an endless one
print(list(islice(count(1), 5)))                  # [1, 2, 3, 4, 5]

# cycle(): repeat a sequence forever
doctors = cycle(["Dr. Rao", "Dr. Iyer"])
for token in islice(count(1), 5):
    print(f"Token {token} goes to {next(doctors)}")
# Token 1 goes to Dr. Rao
# Token 2 goes to Dr. Iyer
# Token 3 goes to Dr. Rao
# Token 4 goes to Dr. Iyer
# Token 5 goes to Dr. Rao

# chain(): join several iterables into one stream
morning = ["Ravi", "Meena"]
evening = ["Arjun", "Fatima"]
for name in chain(morning, evening):
    print(name, end=" ")   # Ravi Meena Arjun Fatima
print()

Be careful: count() and cycle() never stop by themselves. Always limit them with islice(), a break, or something like zip() with a shorter list.

Building a Pipeline of Generators

Generators become really powerful when you connect them, like pipes in a water system. Each generator takes items from the one before it, does one small job, and passes the result on. In the example below, one stage cleans each line, the next turns it into a dictionary, and the last keeps only senior citizens, who get priority at the counter.

def read_lines(lines):
    for line in lines:
        yield line.strip()

def parse(rows):
    for row in rows:
        token, name, age = row.split(",")
        yield {"token": int(token), "name": name, "age": int(age)}

def seniors_only(patients):
    for patient in patients:
        if patient["age"] >= 60:
            yield patient

raw = [
    "1,Ravi,34\n",
    "2,Kamala,67\n",
    "3,Arjun,8\n",
    "4,Joseph,72\n",
    "5,Lakshmi,61\n",
]

# Each stage pulls one item at a time from the stage before it
pipeline = seniors_only(parse(read_lines(raw)))

for patient in pipeline:
    print(f"Priority token {patient['token']}: {patient['name']} ({patient['age']})")

# Priority token 2: Kamala (67)
# Priority token 4: Joseph (72)
# Priority token 5: Lakshmi (61)

No stage builds a full list. When the for loop asks for the next patient, the request travels back through the pipe, and exactly one line is read, parsed and checked. You could replace raw with an open file of ten million lines and the program would still use very little memory. Each small function is also easy to read and test on its own.

Real-Life Example: Tokens for Every Department

A real hospital has several departments, and each needs its own series of tokens: OPD-001, EYE-001, DEN-001 and so on. Here, each department gets its own endless generator, stored in a dictionary. When a patient arrives, we call next() on the right machine. Each generator keeps its own count, separately from the others.

from itertools import count, islice

def department_tokens(prefix):
    """Endless tokens like OPD-001, OPD-002, ..."""
    for number in count(1):
        yield f"{prefix}-{number:03}"

machines = {
    "OPD": department_tokens("OPD"),
    "Dental": department_tokens("DEN"),
    "Eye": department_tokens("EYE"),
}

arrivals = [
    ("Ravi", "OPD"), ("Meena", "Eye"), ("Arjun", "OPD"),
    ("Fatima", "Dental"), ("Joseph", "OPD"), ("Kamala", "Eye"),
]

for name, dept in arrivals:
    token = next(machines[dept])
    print(f"{name:<8} {dept:<7} {token}")

# Ravi     OPD     OPD-001
# Meena    Eye     EYE-001
# Arjun    OPD     OPD-002
# Fatima   Dental  DEN-001
# Joseph   OPD     OPD-003
# Kamala   Eye     EYE-002

# Preview the next three OPD tokens without printing a million
print(list(islice(machines["OPD"], 3)))   # ['OPD-004', 'OPD-005', 'OPD-006']

Try these changes:

1. Add a “Skin” department with the prefix SKN and two new patients.
2. Make the Eye tokens start from 50 by giving department_tokens a second parameter for the starting number.
3. Use cycle() to send OPD patients to two doctors in turn, and print the doctor’s name next to each token.

Common Mistakes

• Using a generator twice. A generator can be used only once. After the last item, it stays empty. If you need the values again, create a new generator, or store them in a list.

tokens = (n for n in range(1, 4))

print(list(tokens))   # [1, 2, 3]
print(list(tokens))   # []  (the generator is already used up)

# Fix: create a new generator each time you need the values again
def make_tokens():
    return (n for n in range(1, 4))

print(list(make_tokens()))   # [1, 2, 3]
print(list(make_tokens()))   # [1, 2, 3]

• Calling len() or using an index on a generator. A generator does not know how many items it will produce, and it cannot jump to item 5. Use sum(1 for _ in gen) to count, or convert it to a list first if it is small.

tokens = (n for n in range(1, 4))
print(len(tokens))   # ERROR: TypeError: object of type 'generator' has no len()

• Turning an endless generator into a list. list(count(1)) never finishes and slowly fills the memory. Always take a limited part with islice().
• Expecting the function body to run when you call it. token_counter() only creates the generator. Nothing inside runs until the first next() or loop.
• Writing return where you meant yield. A return value inside a loop ends the generator immediately. Use yield for each item you want to hand out.

Practice Task: Appointment Slot Generator

The hospital also gives appointment times. Write a generator function appointment_slots(start_hour, end_hour, minutes) that produces times as text. Your solution should:

• Start at start_hour o’clock and add minutes each time.
• Yield each time in the format "09:00", with two digits for hours and minutes.
• Stop before end_hour o’clock.
• Print all slots from 9 to 11 with 20-minute gaps.
• Use islice() to print only the first three slots from 14 to 17 with 15-minute gaps.

Hint: count the minutes from midnight, and use divmod(total, 60) to get hours and minutes.

from itertools import islice

def appointment_slots(start_hour, end_hour, minutes):
    # TODO 1: start at start_hour:00
    # TODO 2: yield each time as a string like "09:00", "09:15"
    # TODO 3: stop before end_hour:00
    pass

# TODO 4: print every slot from 9 to 11 with 20-minute gaps
# TODO 5: use islice to print only the first 3 slots from 14 to 17 with 15-minute gaps

from itertools import islice

def appointment_slots(start_hour, end_hour, minutes):
    total = start_hour * 60          # work in minutes from midnight
    end = end_hour * 60
    while total < end:
        hour, minute = divmod(total, 60)
        yield f"{hour:02}:{minute:02}"
        total += minutes

for slot in appointment_slots(9, 11, 20):
    print(slot)
# 09:00
# 09:20
# 09:40
# 10:00
# 10:20
# 10:40

print(list(islice(appointment_slots(14, 17, 15), 3)))   # ['14:00', '14:15', '14:30']

Key Takeaways

• An iterable is anything you can loop over. An iterator gives one item per next() call and raises StopIteration at the end.
• A for loop is just iter() once, then next() again and again.
• A function with yield is a generator: it pauses at each yield and continues from the same place.
• Generator expressions use round brackets and use far less memory than lists for big data.
• Read big files with for line in f, and connect small generators into pipelines.
• itertools gives you count, islice, cycle, chain and many more ready-made tools.