Local AI, without the overkill
Start with a useful task. Then find the smallest setup that does it well.
A local AI setup can become an infrastructure project before it has answered a single useful question. The model, the interface, the routing layer, the dashboards. All interesting. None of them are the starting point.
The starting point is a task worth repeating.
One task, one baseline
Pick something concrete: summarize a short note, classify a support message, or explain a small function. Keep a handful of representative inputs, including a few awkward ones. They become the baseline when changing models or settings.
Write down what a useful answer looks like. “Seems smart” is hard to compare. “Extracts all three action items without inventing a deadline” is much better.
Treat context as a resource
Longer context is not free. Ollama documents the relationship between context length, parallel requests, and memory requirements. Before increasing every limit, look at what the task actually needs.
A sensible experiment log can be very small:
- Model: exact name and quantization.
- Context: configured context length.
- Input: a fixed example from the test set.
- Result: useful, incomplete, or wrong.
- Latency: time until the response is usable.
The goal is not a public leaderboard. It is a setup that works for the things you do.
Add complexity when it earns its place
Start with one model and one interface. Add routing when there are two distinct workloads. Add tools when an answer needs an action. Keep cloud connections explicit so a local experiment does not quietly change its privacy assumptions.
A smaller setup that gets used is more interesting than a perfect setup that is always being configured.
Next on the workbench: a small evaluation harness for Home AI, with repeatable inputs and a record of what changed.
Reference: Ollama FAQ — memory, context, and concurrency.