Ollama v0.34.2-rc2 Improves mlxrunner Memory Management with KV Buffer Release

Core summary

One sentence to understand this update

Ollama's v0.34.2-rc2 release candidate introduces an optimization in `mlxrunner` to release freed KV buffers every 256 generated tokens during speculative decode, enhancing memory efficiency.

Impact & opportunity

What this could mean

Developers using Ollama, especially with speculative decoding, may experience improved memory utilization and potentially better performance due to this buffer management enhancement.