Kimari Local AI
Run useful local AI on older NVIDIA GPUs
Local-first · Open-source · GGUF runtime · Gateway dashboard · Agent-ready
What is Kimari?
Kimari is an open-source framework for running local language models on consumer NVIDIA GPUs, especially older cards such as the GTX 1060 6GB and GTX 1080 8GB.
Current Status
| Framework / CLI | Usable alpha |
| Local GGUF runtime | Working |
| OpenAI-compatible endpoint | Working |
| GTX 1060 validation | Validated |
| Gateway Dashboard | Local preview |
| Public Kimari-4B weights | Not released |
Quick Start
Requirements: Linux/WSL2, Python 3.10+, CUDA Toolkit, NVIDIA GPU.
curl -fsSL https://raw.githubusercontent.com/smouj/kimari-local-ai/main/install.sh | bash
kimari doctor --deep
kimari pull test
kimari start --profile testVerify: curl http://127.0.0.1:11435/health → {"status":"ok"}
Kimari-4B Model Training
We are iterating on a custom model for Kimari. The focus is benchmark honesty: the model must not fabricate scores or accept unverified claims.
| Phase | Status | Result |
|---|---|---|
| V2 SFT (SmolLM3-3B) | Complete | 1 safety + 2 factual regressions |
| V3 SFT (+12 corrective) | Complete | es-tech-003 fixed, refuse-010 persisted |
| V4 SFT (+6 aggressive) | Complete | Aggregate improved, refusal persisted |
| A/B: Qwen3 vs SmolLM3 | Complete | Qwen3 selected |
| DPO Pilot 15% (Qwen3) | Complete | 3G/0R/7Y — 0 fabrications |
| DPO Full 420 (Qwen3) | Complete | Regressed (1 fabrication) |
| Expanded eval (180 prompts) | In progress | Candidate V5 pending |
Gate: BLOCKED — must reach 0 benchmark fabrications in expanded evaluation.
No public weights, adapters, or GGUF files at this stage.
Documentation
| Project Truth | Honest status of what Kimari is and isn't |
| Getting Started | Manual setup and first launch |
| CLI Reference | Complete command reference |
| Release Gate | Gate states for Kimari-4B |
| DPO Smoke Report | Pipeline validation and engineering lessons |
| Full420 Regression | Why more DPO wasn't better |