Radar Telegram alerts Open Radar

vLLM 0.34.0 changes working with memory by default: CUDA and ROCm include extended KV cache, its size is considered free from memory after warmup, gpu memory utilization is now 1.0. To return the previous check, according to the notes for the release, you can flag -gpu-memory-utilization 0.9, and the cache itself disables - unthinkable-kv-cache. MiniMAX-M3 on the H200 cache increased from 433 to 672 thousand tokens, which increases the throughput at high load. However, marking delay on some models is increased by 6-18%. The default block size has been increased from 16 to 64, the V1 model runner has been removed, and the VLLM USE V2 MODEL RUNNER=0 now drops the launch. You need Face Hug 2.0 and newer. What else can break with the update, colleagues wrote. Channel @neuro.

תרגם לעברית

Нейроканал (RU) · · Source

vLLM 0.34.0 changes working with memory by default: CUDA and ROCm include extended KV cache, its size is considered free from memory after warmup, gpu memory utilization is now 1.0. To return the previous check, according to the notes for the release, you can flag -gpu-memory-utilization 0.9, and the cache itself disables - unthinkable-kv-cache. MiniMAX-M3 on the H200 cache increased from 433 to 672 thousand tokens, which increases the throughput at high load. However, marking delay on some models is increased by 6-18%. The default block size has been increased from 16 to 64, the V1 model runner has been removed, and the VLLM USE V2 MODEL RUNNER=0 now drops the launch. You need Face Hug 2.0 and newer. What else can break with the update, colleagues wrote. Channel @neuro.

More on Radar

Want to see every report in real time?

Hundreds of sources worldwide, translated into your language, with alerts for what matters to you.

Sign up free
Share on WhatsApp Get alerts on Telegram