itch.io

bartowski/gemma-2-9b-it-GGUF at main The 8_0.gguf or 8_0_L.gguf actually fit entirely in 12GB and are really good at following instructions. Kobold should set it to fully offload to GPU automatically, and you can crank the context size to 8192 in the Koboldcpp UI. Might still need to fiddle with the system prompt in the Formamorph settings. Also, by default the "Endpoint URL" for Formamorph should be "http://localhost:5001/v1/chat/completions".

You'll probably run out of context if you play for a while but that should be enough to get you going for starters.