Generator Exceptions Surface on the First next(): try Around create_chat_completion(stream=True) Never Sees the Context Window Error

In Python, calling a function that contains yield gives you a generator object; not a line of the body has run yet. Any raise in the body also waits for the first next() (or the first value a for loop pulls). So if a try wraps the call but not the iteration, exceptions raised in the body never pass through that try.

Whether a library function is a generator isn’t visible at the call site. llama-cpp-python’s create_chat_completion is an ordinary function, but with stream=True it returns the internal _create_completion generator, and both tokenizing the prompt and the context window check live in that generator’s body.

01The code

This is how Autarkia streams from a local GGUF model (unrelated branches removed):

try:
    stream = llama_model.create_chat_completion(
        messages=messages, max_tokens=max_tokens, stream=True,
    )
except Exception as e:
    if is_context_window_error(str(e)):
        # 解析 "Requested tokens (N) exceed context window of M"，
        # 收缩 max_tokens、裁剪 messages 后重试
        stream = self._retry_context_window_stream(messages, kwargs, max_tokens, str(e))
    ...

self._iterate_stream(stream, ...)   # 里面是 for chunk in stream

It reads sensibly: send the request, and if it overflows the window, shrink and retry. With the real library, though, create_chat_completion(..., stream=True) just builds the generator and returns normally, however long the prompt is. The error is raised when for chunk in stream in _iterate_stream pulls its first value, lands in the except Exception there, and is reported as a GPU streaming error. No retry happens.

The relevant structure in llama-cpp-python 0.3.35, simplified:

def _create_completion(self, prompt, max_tokens, stream, ...):
    prompt_tokens = self.tokenize(...)
    if len(prompt_tokens) >= self._n_ctx:
        raise ValueError(
            f"Requested tokens ({len(prompt_tokens)}) exceed context window of {self._n_ctx}"
        )
    ...
    yield ...                      # 有 yield，整个函数是生成器

def create_completion(self, ..., stream=False):
    completion_or_chunks = self._create_completion(...)
    if stream:
        return completion_or_chunks   # 原样返回，函数体还没运行
    return next(completion_or_chunks) # 非流式在这里就会抛

The non-streaming call does one next() inside create_completion, so the same try works on a non-streaming path. Autarkia’s non-streaming generator is written that way and is fine; only the streaming one is affected.

I couldn’t load the library on this dev machine (it’s a CUDA build and the runtime is missing), so the conclusion comes from the installed version’s source: ast confirms _create_completion has a yield in its body. I also wrote a minimal reproduction of the three layers without the library; the test double went into the retry branch, and the real structure raised during iteration.

02Why it went unnoticed

The fake model in the unit tests looks like this:

class _FakeLlamaModel:
    def create_chat_completion(self, messages=None, **kwargs):
        self.calls.append((messages, kwargs))
        if self._chat_error_once is not None:
            err, self._chat_error_once = self._chat_error_once, None
            raise err                       # 调用时就抛
        return iter(list(self._chunks))

It’s an ordinary function that raises on the call, more eager than the real library. So the "retry succeeds after a context window error" test passed and every line of the retry branch was executed. When this file’s tests were filled out in September, coverage went from 41% to 100%, and the same pass found and removed a different branch in this file that could never run. This branch wasn’t suspected because coverage said it had run.

The logs vouched for it too. In the few days of Autarkia logs still on disk, "request exceeded the context window, shrinking and retrying" appears 1,028 times, which looks like the mechanism working. Looking at the parameters, there are only two variants, requested=140, window=128 and requested=300, window=128, both literals from the tests; the unit tests write into the same log directory.

Before that, when an old inference adapter was deleted in June, one of the reasons given was that its retry had been replaced by this function, and an analysis doc from November still says an error is returned only if the retry fails. Everyone believed it worked. On top of that, the path is only taken with a local GGUF model and without the C++ scheduler, and since late October the production models have all been cloud ones, so few requests go through it. Whether a real request ever hit it I can’t tell: older logs are gone.

03It wouldn’t have helped anyway

Reading further into the retry function, it probably wouldn’t have helped even if called. The requested in the error is just the prompt’s token count; if max_tokens is too large the library trims it to the remaining space instead of raising. The retry nevertheless subtracts the overflow from max_tokens. The only part that matters is the message trimming done alongside, and it trims to window * 2 characters; by the repo’s own estimate of about 1.67 tokens per Chinese character, that’s still more than three times the window (I haven’t measured the ratio with the real tokenizer). There’s also a "halve it if shrinking didn’t help" branch whose condition only holds when max_tokens <= 16, and once inside the result is always 16. The test is named shrinks_and_halves and asserts the same value that was passed in.

04Fixing it

The most direct fix is to pull the first chunk inside the same try, so the generator has actually started before it’s handed on:

import itertools

try:
    stream = llama_model.create_chat_completion(messages=messages, stream=True, ...)
    first = next(stream)            # 长度检查在这一步执行
except StopIteration:
    stream = iter(())
except Exception as e:
    ...                             # 这里才真的接得到超窗口错误
else:
    stream = itertools.chain([first], stream)

Sturdier still is not to rely on the error at all: measure the prompt with the model’s own tokenize before the call and trim by tokens to fit the window. Autarkia’s scheduler path already has a token-based trimming function the streaming GGUF path can reuse. The test double should also become a generator function that raises on the first next(), as lazy as the real library; otherwise the next rewrite will slip past the same tests.

All of this is still on HEAD; the fixes are noted for the main project.