What We Mean When We Say a Generator Is Lazy

Is your streaming endpoint writing tokens nobody reads?

I agree to the Privacy Policy Review Your Streaming Endpoint

A generator does no work until someone asks for the next value. In a streaming endpoint, the one asking is not the reader.

Ask ChatGPT a question and watch the answer. It does not appear all at once. It appears a few words at a time, as if someone were typing on the other side.

Nobody is typing. Here is what is actually happening, in three steps.

1. The AI writes its answer one small piece at a time. A piece is usually a word, or part of a word. Engineers call it a token.
2. Between you and the AI sits a server: a program that takes your question, asks the AI, and sends the answer back to your phone.
3. The server does not wait for the whole answer. The moment the AI finishes a token, the server passes it on to you. That is why the words trickle in. This is called streaming.

One more thing worth knowing: companies pay for AI by the token. Every token the AI writes costs a little money, whether anyone reads it or not.

In Python, the part of the server that passes tokens along is almost always a generator. The name sounds technical. The idea is not.

Picture two restaurants.

The first is a buffet. All the food is cooked before you arrive and put on the table. Eat one plate and leave, and the rest goes in the bin. In Python, that is a list: everything made up front.

The second has a chef who cooks one dish at a time, and starts the next one only when you finish the one in front of you. Stop eating, and the chef stops cooking. Nothing is wasted. That is a generator.

Here it is in code. Skip the syntax if you like; just read what it prints.

def courses():
    for dish in ["soup", "fish", "cake"]:
        print(f"  cooking {dish}")
        yield dish

menu = courses()
print(next(menu))

  cooking soup
soup

We asked for one dish. The chef cooked soup, and only soup. The fish and the cake were never made. That is what programmers mean when they say a generator is lazy: it does no work until someone asks for the next thing. Here, lazy is a compliment.

It is also why streaming servers are usually written with one. Ask the AI for the next token, send it to the reader, ask for the next one. In code it is about four lines:

async def handler(reader, writer):
    async for token in model.stream(prompt):   # for every token the AI writes
        writer.write(event(token))             # send it to the reader
        await writer.drain()                   # wait until it has gone out

So we reach the sensible conclusion: if you stop reading, the chef stops cooking, and nobody pays for words nobody read.

This article is about why that sensible conclusion is wrong, and what it costs.

Who is asking

In the restaurant, the chef can see your table. When you stop eating, the chef notices, because you stop asking for more.

Now move the kitchen across town. The chef is the AI, on a server somewhere. You are on your phone. Halfway through a long answer you get distracted, lock the phone and put it in your pocket. The app is still connected. You are just not reading any more.

The answer was going to be about 2,000 tokens, a few pages of text. You saw 40. How many tokens does the AI write?

Common sense has an answer ready. The generator is lazy. You stopped asking, so it stops writing. Around 40, maybe a few more.

Twenty-eight thousand tokens nobody read

We do not have to guess. We can test it on one laptop. One program plays the AI: it writes tokens as fast as it can. Another plays your locked phone: it connects, and then never reads a thing. Then we count how many tokens the AI writes before anything slows it down.

For the curious: each token goes out as a 209-byte message, the same shape real AI services use.

import asyncio, json, socket

def event(tok: str) -> bytes:
    chunk = {"id": "chatcmpl-9x7Qp2mZk3", "object": "chat.completion.chunk",
             "created": 1759132800, "model": "some-model-2026-09-01",
             "choices": [{"index": 0, "delta": {"content": tok},
                          "finish_reason": None}]}
    return f"data: {json.dumps(chunk)}\n\n".encode()

async def silent_reader():
    produced, stalled = 0, asyncio.Event()

    async def model():
        while True:
            yield " the"

    async def handler(reader, writer):
        nonlocal produced
        async for tok in model():
            writer.write(event(tok))
            try:
                await asyncio.wait_for(writer.drain(), timeout=1.0)
            except TimeoutError:        # a full second with no room to write
                stalled.set()
                return
            produced += 1

    server = await asyncio.start_server(handler, "127.0.0.1", 0)
    port = server.sockets[0].getsockname()[1]
    _r, w = await asyncio.open_connection("127.0.0.1", port)   # never read
    await stalled.wait()
    w.close(); server.close()
    return produced

print(asyncio.run(silent_reader()))

28573

In plain words: you read 40 tokens. The AI wrote 28,573. Someone paid for all of them.

The generator did not stop at 40. It ran on for 28,573 tokens before anything pushed back, the same number in five runs out of five.¹ At a brisk 50 tokens a second that is nine and a half minutes of writing, or fourteen complete 2,000-token answers. The real answer finished long before the generator ever paused. We paid for all of it. The reader saw forty words.

The belt does not watch the seats. The kitchen keeps its own pace. Photo by Ice Tea on Unsplash.

We cannot accept both halves of that. Either the generator is lazy, or a reader who stops reading costs nothing. The run says the second one is false, and the generator never stopped being lazy. It did exactly what it promised: no work until someone asked. So someone asked. Who?

The obvious sentence carried two assumptions nobody stated. The first is that the reader is the one calling for the next value. Across a socket, it is not. Our async for loop calls for the next token, and its only brake is drain(), which waits only when there is no room left to write. The second is that a value handed over is a value taken. Handing a token to the socket means handing it to a buffer, and a buffer accepts it whether anyone is at the other end or not.

So “lazy” was being said in two ways. There is lazy in the producer: it does no work until asked. And there is paced by the consumer: it does no more work than the consumer uses. Inside one process these are one property, and nobody needs two words for it. Across a socket they come apart. Pull becomes push the moment a value crosses a buffer.

Laziness belongs to the producer. Pace belongs to the whole path, and the path is mostly buffers.

How deep the buffers are

Let us count the buffers, because the number above should not be a mystery. It should be something we can predict with a pen.

A token leaves our loop and waits in three places. First, asyncio’s own write buffer in our process, which lets drain() return immediately until it holds about 64 KiB. Second, the kernel’s send buffer on our side. Third, the kernel’s receive buffer on the reader’s side. The reader not reading only reaches our loop once all three are full. So:

tokens written before the reader is felt
    = (write buffer + send buffer + receive buffer) / bytes per token

Nothing in that line is invented. We can ask the sockets for all three sizes at the moment of the stall:

user_space = writer.transport.get_write_buffer_size()                    # 65,624
sndbuf = server_sock.getsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF)     # 4,194,304
rcvbuf = client_sock.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)     # 1,665,640

That is 5,925,568 bytes. Divided by 209 bytes per event, it predicts 28,352 tokens. We measured 28,573, which is 0.8% more.

Capacity that sits between the producer and the person. Nothing upstream feels the far end until every tank is full. Photo by Julia Taubitz on Unsplash.

Now the second number, which is the one that matters. A long answer is 2,000 tokens, or 418,000 bytes in these events. The kernel’s send buffer on our side holds 4,194,304 bytes by itself. So a whole answer fits ten times over in the first buffer the reader cannot see, before the reader’s side is even counted. For a text stream, backpressure cannot reach the model. The pipe is deeper than the answer is long.

This changes what we can build. The easy case turns out to be fine. We ran it too: a model at 50 tokens a second, a reader who reads 40 tokens and then closes the connection. Our loop wrote one more token, got a ConnectionResetError 41 milliseconds later, and the generator’s finally block, where the upstream connection to the model would be closed, ran in the same 41 milliseconds. Five runs, same result. A reader who leaves is cheap.

The expensive reader is the one who goes quiet without leaving. TCP gives us no signal for them until the buffers fill, and we just measured how long that takes. So the brake has to be one we build ourselves:

from contextlib import aclosing

async def handler(reader, writer):
    async with aclosing(model.stream(prompt, max_tokens=1000)) as stream:
        async for token in stream:
            writer.write(event(token))
            await writer.drain()

Two things are in that loop that the obvious version did not have. A max_tokens cap, because it is the only limit that holds when nobody is reading. And aclosing, which closes the generator, and whatever upstream connection its finally holds, when we leave the loop for any reason. In our runs CPython happened to close it promptly anyway. We should not build on something that merely happens. If the product really needs the reader’s pace, the reader has to report it: the client sends back what it has shown, and the server stops when the gap grows. That brake is application code, because nothing lower down will provide it.

Reporting · TechCrunch · June 2026

The token bill comes due: Inside the industry scramble to manage AI's runaway costs

Those questions are asked at the level of budgets and teams. The tokens in this article are on the same bill, one loop further down. There, the token control is the one we just wrote.

"Now the conversations are about, 'hey, we're spending so much. What visibility do you have? What auditability do you have? What token controls do you have? What is the efficiency of your models?'"

— Alexander Embiricos, OpenAI's head of enterprise, to TechCrunch

Read the TechCrunch report

Source card image: photo by Towfiqu barbhuiya on Unsplash.

What the buffer model hides

The model is one hop on one machine. A real response passes through more: a reverse proxy, perhaps a CDN, a mobile network, the phone’s own receive window. Each one adds depth, and none of them removes any. So the 28,573 is probably low for a real deployment, not high. What we can claim is the order of magnitude, not the digit.

It is also less exact than the 0.8% suggests. We ran the same experiment with bare text instead of framed events, about five bytes per token. The buffers predict 1,194,300 tokens. We measured 1,055,528, which is 12% short. The reason is that the kernel budgets memory, not payload. Every write carries some bookkeeping, and a five-byte write carries more of it per byte than a 209-byte one does. We did not measure which overhead it was. The 0.8% agreement for framed events is closer than the model deserves, and this run is the one that says so.

There is also a boundary Python cannot cross. When our generator stops, it can close the connection to the model provider. Whether the provider then stops generating, and stops billing, is the provider’s decision, not ours. We should check it before we trust it.

And the model hides one more thing, the same thing the obvious sentence hid, only one level further down. It makes the socket visible, and so it treats a token the phone accepted as a token delivered. A browser that renders every token has still not shown us a person reading it. The buffer model moved the line from “handed to the socket” to “accepted by the phone”. Where the answer meets a person, it has not moved the line at all.

The kitchen across town

Go back to the kitchen that cooks one course at a time. It is still lazy. It still starts the next course only when asked.

What changed is who asks. When the kitchen moved across town, the diner stopped asking and a courier took over. The courier leaves each dish on the doorstep, comes back for the next, and stops only when the doorstep is full. We measured the doorstep. It holds fourteen dinners.

A generator is lazy with respect to whoever calls next. In a streaming endpoint, that caller is our own loop, not the reader. If we want the reader to set the pace, we have to build that ourselves. The socket will not.

References

1. Python documentation, StreamWriter.drain, Streams, asyncio
2. Python documentation, contextlib.aclosing
3. Yury Selivanov, PEP 525: Asynchronous Generators, 2016
4. Linux man-pages, tcp(7), tcp_wmem and tcp_rmem
5. Rebecca Bellan, The token bill comes due: Inside the industry scramble to manage AI’s runaway costs, TechCrunch, 5 June 2026

Notes

Keep exploring

For the difference between waiting and working in Python, read Python Concurrency: Threads, Processes, and the Shape of Waiting. For why a model writes one token at a time in the first place, read What We Mean When We Say the Model Thinks Step by Step.

Hero photo: MealPro on Unsplash. Article photos: Ice Tea, Julia Taubitz and Towfiqu barbhuiya on Unsplash.

Part of the "primitives first, then compositions" series.

Is your streaming endpoint writing tokens nobody reads?

I agree to the Privacy Policy Review Your Streaming Endpoint