Shihao Qi
← Notes

json.dump iterates while it writes: save outside the lock and you get “dictionary changed size during iteration”, save inside and every writer waits

3 min read#Python#Concurrency

json.dump(data, f) doesn’t serialize first and then write; it uses the pure-Python encoder and writes while iterating, so serialization and disk I/O can’t be separated. Inside a lock, writers wait as long as the disk takes; outside, any concurrent change to the dict raises RuntimeError. Autarkia’s memory save went back and forth between the two.

Table of Contents

json.dump(data, f) and json.dumps(data) don’t use the same encoder. json.dumps builds the whole string at once, and on my Python 3.13 it uses the C implementation. json.dump writes to the file in chunks through a pure-Python generator that writes as it iterates. That means the first chunk is hitting the disk before the dict has been fully traversed, and other threads have every opportunity to modify it meanwhile.

With a background thread adding and removing keys, 200 runs each on the main thread:

python
json.dump(d, SlowFile(), indent=2)   # 200/200 次 RuntimeError
json.dumps(d, indent=2)              # 0/200

SlowFile.write contains just time.sleep(0), standing in for the GIL release a real disk write causes.

01First: writing to disk inside the lock

Autarkia’s memory manager guards several large in-memory dicts with a read-write lock. In April, the background save thread held the write lock across the whole save, serialization plus writing several JSON files, and kept it for over ten seconds while every read and write queued. The fix moved the save outside the write lock, leaving only queue clearing and index updates inside.

02Second: iterating outside the lock

In September the save thread began occasionally raising RuntimeError: dictionary changed size during iteration. The save function no longer held any lock, and the disk write did this:

python
with open(temp_path, "w", encoding=encoding) as f:
    json.dump(data, f, ensure_ascii=False, indent=2)

The first round of fixes wrapped dict iteration in list(...) snapshots inside the callees. list(d.items()) is a one-shot C-level copy that the GIL doesn’t interrupt, so those spots did stop raising. But the iteration inside json.dump wasn’t covered by any of those snapshots; it was still serializing the original dict.

The second round put the whole save function back under a lock, a read lock this time:

python
def safe_save_all(manager):
    with get_read_lock(manager):
        ...
        manager._safe_json_dump(short_disk_records, short_file)
        manager._save_weighted_data_locked()
        manager._save_important_prompts_locked()
        manager._write_readable_history_mirror()
    manager.last_save_time = time.time()

The read lock keeps writers out, so iteration can’t be interrupted anymore. But that block also writes several files and fsyncs them. Readers can still come in; writers wait until the whole save hits disk. Half of what April moved out of the lock came back. How long writers wait now, I haven’t measured.

03Both times the same thing moved

Both fixes (both mine) were about “which section should this lock cover”, without noticing that json.dump makes “iterate the data” and “write to disk” a single action that can only enter or leave the lock together. Split them and the lock boundary becomes obvious:

python
with get_read_lock(manager):
    payload = json.dumps(data, ensure_ascii=False, indent=2)   # 只在内存里，很快
atomic_write_text(path, payload)                               # 锁外写盘

Only in-memory work happens under the lock, producing a snapshot nobody else can modify; the disk write happens outside. If the data holds custom objects that make json.dumps call back into Python, copy.deepcopy it under the lock first. I haven’t landed this in the main project yet; it’s in my list of fixes. The comment in that code needs correcting too: it says the call that raises is json.dumps(indent=...), while on 3.13 the one that actually breaks is json.dump(..., f).

Related

Filtering kwargs with inspect.signature: a function that takes **kwargs looks like it accepts nothing, and a retry lost its model_path2026-11-08inspect.signature(f).parameters lists only declared parameters; **kwargs shows up as a single entry named kwargs. Checking by name whether a callee accepts an argument lets nothing through to the most permissive function. Autarkia’s empty-reply retry lost its model_path this way and fell back to the default model; the resulting 400s landed among upstream 503s, and that day’s error triage filed them as gateway noise.#Python#LLM#DebuggingGenerator Exceptions Surface on the First next(): try Around create_chat_completion(stream=True) Never Sees the Context Window Error2026-11-08Calling a generator function only builds a generator; the body, including any raise in it, runs on the first next(). llama-cpp-python checks the context window inside such a body, so the try around a streaming call, and the shrink-and-retry under it, never fire with the real library. The test double raised at call time, so the tests passed with 100% coverage.#Python#LLM#Error Handling#Testing#AutarkiaLLM Request Observability: Logs Count Messages, and the Native Tools Schema Shows Up Nowhere2026-11-07With native function calling, tool definitions go in the tools parameter, not in messages. A log that measures message length can’t see them, and streaming responses omit usage by default. Autarkia sent 82 tools and 47,898 characters of schema every turn while the prompt log showed about 2,200 characters; the API counted 21k–23k tokens.#LLM#Python#Observability#Autarkia
← Back to notes