TAG
LLM
Everything filed under this tag.
10 entries01Writing
Filtering kwargs with inspect.signature: a function that takes **kwargs looks like it accepts nothing, and a retry lost its model_path2026-11-08inspect.signature(f).parameters lists only declared parameters; **kwargs shows up as a single entry named kwargs. Checking by name whether a callee accepts an argument lets nothing through to the most permissive function. Autarkia’s empty-reply retry lost its model_path this way and fell back to the default model; the resulting 400s landed among upstream 503s, and that day’s error triage filed them as gateway noise.#Python#LLM#DebuggingGenerator Exceptions Surface on the First next(): try Around create_chat_completion(stream=True) Never Sees the Context Window Error2026-11-08Calling a generator function only builds a generator; the body, including any raise in it, runs on the first next(). llama-cpp-python checks the context window inside such a body, so the try around a streaming call, and the shrink-and-retry under it, never fire with the real library. The test double raised at call time, so the tests passed with 100% coverage.#Python#LLM#Error Handling#Testing#AutarkiaLLM Request Observability: Logs Count Messages, and the Native Tools Schema Shows Up Nowhere2026-11-07With native function calling, tool definitions go in the tools parameter, not in messages. A log that measures message length can’t see them, and streaming responses omit usage by default. Autarkia sent 82 tools and 47,898 characters of schema every turn while the prompt log showed about 2,200 characters; the API counted 21k–23k tokens.#LLM#Python#Observability#AutarkiaThe Message You Edited Is Still in Memory2026-11-01After a conversation forks, filtering raw messages by branch is easy, but summaries and facts still remember the abandoned line. Feeding Mem0 only the current branch’s messages lowered its leak rate from 60.8% to just 50.0%.#Memory#Evaluation#Measurement#LLMThe Synthetic Set Picked C. Forty-Four Real Questions Picked B.2026-10-27A synthetic evaluation set gave a clean answer to what granularity memory should be stored at. Real questions from real chats gave a different one — and along the way turned up three measurement problems.#Memory#Evaluation#Measurement#LLMWhere the Prefix-Cache Hits Went2026-10-27The hit rate dropped, and the total alone couldn’t say why. Looked at in 128-token blocks, the losses came from one section placed in the wrong spot, and from two providers with completely different caching rules.#LLM#Cost#ObservabilityWhy the More You Write a Persona, the Less Human It Sounds2026-10-21The persona grew from 800 to 8,000 characters, and replies started sounding like the character reciting its own spec. Most of the problems turned out to be in the code that assembles the prompt, not in the persona text itself.#Prompt#Persona#LLMOne pop() in a subclass payload builder: max_tokens deleted before sending, invisible to callers, logs and tests2026-10-21The parent class builds the request body, the subclass pops a field before returning it, and whatever the caller passes stops mattering without any error. Autarkia’s MiniMax client has deleted every caller’s max_tokens since the first commit, leaving a reasoning model with no output ceiling, and it was mentioned three times in my notes without being changed.#Python#LLM#ObservabilityTreat the LLM as an Untrusted External Service2026-10-18It behaves like a remote service: it times out, it returns nothing, it mixes its reasoning into the body, and it fills in a number by feel.#LLM#Engineering