There’s a little known feature in the closed-source NVIDIA driver that lets you freeze a running CUDA process, serialize its GPU state into host memory, and later restore it to the GPU exactly as it was. We used it in an earlier post to speed up SGLang server startup by up to 70x. The utility is called cuda-checkpoint. The feature is documented, but how it works isn’t. One very frustrating aspect, that dogs anyone trying to use it to checkpoint complex GPU processes, is that the checkpoint trans...