vLLM 0.34.0 changes working with memory by default: CUDA and ROCm include extended KV cache, its size is considered free from memory after warmup, gpu memory utilization is now 1.0. To return the previous check, according to the notes for the release, you can flag -gpu-memory-utilization 0.9, and the cache itself disables - unthinkable-kv-cache. MiniMAX-M3 on the H200 cache increased from 433 to 672 thousand tokens, which increases the throughput at high load. However, marking delay on some models is increased by 6-18%. The default block size has been increased from 16 to 64, the V1 model runner has been removed, and the VLLM USE V2 MODEL RUNNER=0 now drops the launch. You need Face Hug 2.0 and newer. What else can break with the update, colleagues wrote. Channel @neuro.