Shihao Qi
← Writing

MagicMock accepts any arguments: the caller passes one too many, the test passes, production raises TypeError and except swallows it

4 min read#Python#Testing

A MagicMock without a spec doesn’t check call signatures; when the code under test passes the wrong arguments, the double still returns whatever you configured. In Autarkia this meant a block of prompt context was never injected: the real method took only self, the caller passed a user_id, the test double accepted it, and in production the TypeError was swallowed by an outer except.

Table of Contents

On a MagicMock() every attribute exists, every attribute is callable, and any arguments are accepted. Set a return_value and that’s what you get back. So as long as the code under test is calling a double, “wrong arguments” simply can’t happen in tests. create_autospec or patch(..., autospec=True) builds the double from the real object’s signature and raises TypeError on a mismatch:

python
from unittest.mock import MagicMock, create_autospec

class DailyManager:
    def get_today_summary(self):
        ...

m = MagicMock()
m.get_today_summary.return_value = "摘要"
m.get_today_summary("user-1")          # 返回 "摘要"

a = create_autospec(DailyManager, instance=True)
a.get_today_summary.return_value = "摘要"
a.get_today_summary("user-1")          # TypeError: too many positional arguments

01A context block that never reached the prompt

When Autarkia builds a prompt it includes a “today’s summary” section. The caller looked like this:

python
try:
    from core.services.daily.manager import get_daily_manager

    daily_sum = get_daily_manager().get_today_summary(user_id)
    if daily_sum:
        txt += f"\n【今日摘要】\n{daily_sum}\n"
        ...
except Exception:
    pass

get_today_summary has the signature (self) and takes no user_id. Every call raised TypeError, every time except Exception: pass swallowed it, and the section never once appeared in the prompt. The model just seemed not to know about it, with no error anywhere. The line was already like this when this directory first entered git.

The test looked like this:

python
daily_manager = MagicMock()
daily_manager.get_today_summary.return_value = "【今日画像】\n- ..."

with patch(
    "core.services.daily.manager.get_daily_manager",
    return_value=daily_manager,
):
    text = build_user_bio_context_for_chat("test-user")

assert "【今日摘要】" in text

The double accepted the extra user_id, returned the configured summary, and the assertion passed. What the test guaranteed was “if the summary is obtained, it’s assembled correctly”, and the premise, obtaining the summary, was exactly the step that failed in production.

02How it was found

Later a real-data case was added to the same verification script, and the section was missing from the real output. Even then the script only printed a [WARN], because the same information was also injected through another path that called the method without arguments, so production didn’t lose it entirely. And the script is run by hand, outside CI’s pytest collection, so the [WARN] was only visible to whoever ran it.

The fix removed the extra argument, turned the [WARN] into a hard assertion, and added logger.warning(..., exc_info=True) to the except around such context blocks. Two months earlier the diary tool had the identical problem: it called get_entries(dt, scope=scope) against a (self, date) signature, and that time it surfaced only because the user saw a “failed to read diary” message.

03In the repository now

I counted 279 bare MagicMock() calls in the test directory and only 3 with spec or autospec. Not all are risky: if the double stands in for an external service whose call shape is fixed by a protocol, it matters little. The dangerous ones stand in for the project’s own classes, where caller and callee evolve separately.

The fix I’ve noted for the main project: doubles for the project’s own classes always use create_autospec(RealClass, instance=True); new bare MagicMock() calls fail a check, with existing ones whitelisted for now. Any block wrapped in except Exception needs at least one test that reaches the real object, because in that combination a signature error has no visible symptom in production.

Related

Generator Exceptions Surface on the First next(): try Around create_chat_completion(stream=True) Never Sees the Context Window Error2026-11-08Calling a generator function only builds a generator; the body, including any raise in it, runs on the first next(). llama-cpp-python checks the context window inside such a body, so the try around a streaming call, and the shrink-and-retry under it, never fire with the real library. The test double raised at call time, so the tests passed with 100% coverage.#Python#LLM#Error Handling#Testing#Autarkiatime.strftime uses the system time zone: write with the configured zone, read with the system one, green locally and a day off on UTC CI2026-10-17The project had a helper that takes today’s date in the configured time zone, and scattered time.strftime("%Y-%m-%d") calls next to it. On a dev machine whose zone matches the config, both always agree; CI runs in UTC, and between midnight and 8 a.m. Beijing time the reading side goes looking for yesterday’s file.#Python#Time#TestingFiltering kwargs with inspect.signature: a function that takes **kwargs looks like it accepts nothing, and a retry lost its model_path2026-11-08inspect.signature(f).parameters lists only declared parameters; **kwargs shows up as a single entry named kwargs. Checking by name whether a callee accepts an argument lets nothing through to the most permissive function. Autarkia’s empty-reply retry lost its model_path this way and fell back to the default model; the resulting 400s landed among upstream 503s, and that day’s error triage filed them as gateway noise.#Python#LLM#Debugging
← Back to writing