CMake set(CACHE ... FORCE) Overrides -D: -DGGML_CUDA=ON Never Takes Effect

2026-11-102 min read#CMake #C++#Autarkia

set(... CACHE ...) with FORCE overwrites the cache on every configure, including values passed with -D. One FORCE line in CMakeLists, commented as off by default, kept llama.cpp building as CPU-only on Windows: no errors, just slow.

In CMake, set(VAR value CACHE BOOL "" FORCE) overwrites the cached value on every configure, including whatever was passed with -D. For off by default but switchable from the command line, use set(... CACHE ...) without FORCE, or option().

On August 19, GPU inference through Autarkia’s C++ scheduler timed out waiting for the first token. I suspected the virtual environment first, and eventually found two lines in llama.cpp’s startup log: backend_ptrs.size()=1, and every KV cache buffer on dev=CPU. The loaded extension had no CUDA in it at all.

cmake

# Windows 侧 CUDA 编译有兼容性问题，默认关闭
if(WIN32)
    set(LLAMA_CUDA OFF CACHE BOOL "" FORCE)
    set(GGML_CUDA OFF CACHE BOOL "" FORCE)
endif()

This dated from January, when nvcc on Windows choked on a project path with spaces and CUDA was switched off for the time being. The comment says off by default, but FORCE means overwrite the cached value on every configure, including whatever -D passed on the command line. So it isn’t a default, it’s a forced value: on Windows, whatever you pass, you get a CPU build.

What three spellings produce with -DGGML_CUDA=ON on the command line:

cmake

set(GGML_CUDA OFF CACHE BOOL "" FORCE)   # -> OFF
set(GGML_CUDA OFF CACHE BOOL "")         # -> ON
option(GGML_CUDA "build with CUDA" OFF)  # -> ON（不传参数时是 OFF）

It stayed hidden because the result worked: the extension imported, the log said scheduler_py was found, inference produced output, just slowly. After January, CUDA is broken on Windows was the assumption, so a CPU build looked expected. The only evidence was those two device lines, which don’t look like errors and don’t draw a second glance.

The fix turns the forced value into a real default: wrap it in if(NOT DEFINED GGML_CUDA) so a command-line value wins. Writing option() directly is plainer. A separate script now builds the CUDA version into its own directory, and an environment variable picks the backend at startup.

One gap remains: the backend is chosen by directory name, so if build/cuda/ exists it is assumed to be CUDA, and nothing checks after loading. I plan to log the backends actually available once the extension loads, and warn when the config asks for GPU layers but only a CPU backend is present.

Generator Exceptions Surface on the First next(): try Around create_chat_completion(stream=True) Never Sees the Context Window Error2026-11-08Calling a generator function only builds a generator; the body, including any raise in it, runs on the first next(). llama-cpp-python checks the context window inside such a body, so the try around a streaming call, and the shrink-and-retry under it, never fire with the real library. The test double raised at call time, so the tests passed with 100% coverage.#Python#LLM#Error Handling#Testing#Autarkia LLM Request Observability: Logs Count Messages, and the Native Tools Schema Shows Up Nowhere2026-11-07With native function calling, tool definitions go in the tools parameter, not in messages. A log that measures message length can’t see them, and streaming responses omit usage by default. Autarkia sent 82 tools and 47,898 characters of schema every turn while the prompt log showed about 2,200 characters; the API counted 21k–23k tokens.#LLM#Python#Observability#Autarkia Breaking Out of an Async Generator Triggers aclose(): Code After the Last Yield Never Runs2026-11-07The consumer breaks on the last event, the async generator gets aclose(), GeneratorExit is thrown into the suspended yield, and the create_task after it never runs while a finally would. Autarkia’s conversation-history save never ran once because of it; the pattern was fixed in April and reintroduced in August for a reasonable-sounding reason.#Python#asyncio#Streaming#Autarkia

← Back to notes

CMake set(CACHE ... FORCE) Overrides -D: -DGGML_CUDA=ON Never Takes Effect

Related