dipampaul17/KVSplit: Run larger LLMs with longer contexts on Apple Silicon by … – GitHub

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys …
View full source