itch.io

bartowski/gemma-2-9b-it-GGUF at main The 8_0.gguf or 8_0_L.gguf actually fit entirely in 12GB and are really good at following instructions. Kobold should set it to fully offload to GPU automatically, and you can crank the context size to 8192 in the Koboldcpp UI. Might still need to fiddle with the system prompt in the Formamorph settings. Also, by default the "Endpoint URL" for Formamorph should be "http://localhost:5001/v1/chat/completions".

You'll probably run out of context if you play for a while but that should be enough to get you going for starters.

Home · LostRuins/koboldcpp Wiki · GitHub This is the guide for Koboldcpp, which is the version that lets you run AI models. It also sets up an API you can point Formamorph at. For more specific info it's a bit more complicated since getting a model to actually run well on your own computer depends a ton on how much VRAM you have.

For the response_format you have to give it a whole json schema, which is a lot of work. It's just worth it for stuff like this, if you have the time for it. If you can't then that's probably not too bad either tbh, I've been using the Gemma 9B Instruct model with "Include no text aside from the JSON response. Do not wrap the JSON in quotes or markdown." added to the end of the prompt and it actually does it correctly 99% of the time.

The OpenAI API and a lot of the free tools that implement it support a thing called "structured output". See Structured Outputs - OpenAI API. If you use this to define what the response JSON should look like, you should be able to focus your prompts on other things rather than just on the data structure and you can also guarantee that the model returns valid JSON, which would fix a lot of the connection issues and let people use smaller models.

Just something to consider if you haven't already poked at it.