Most tool debates start with the model. I get better results starting with the harness.
By harness I mean the thing wrapped around the model: how it reads the repo, how it applies edits, how it runs tests, what it hands back when it's done. A capable model inside a sloppy harness still produces changes I can't trust. I've watched that happen enough times that harness choice is now the first decision, and model choice is the second.
What I evaluate a harness on
Four things, and if any one of them is weak the model doesn't rescue it.
Context hygiene. Can I keep what the assistant sees tight and repeatable? The harness needs to read the project's rules file on every session without me pasting it. On this site that file is CLAUDE.md, and on my other repos it's usually AGENTS.md next to it.
Change control. Are the edits easy to review and hard to sprawl? I want a diff that matches the task I described, not a diff that also "cleaned up" three unrelated files.
Verification loop. Can the harness run the tests and report the result in the same session? If I have to leave the tool to find out whether the change works, I'll eventually skip that step, and so will anyone else.
Handoff quality. When it's done, do I get a summary I could paste into a pull request? Changed files, what was verified, what wasn't.
The harnesses I actually use
I use four, and I use them for different things. This isn't a ranking.
Claude Code is where I go when I need the codebase understood before anything gets touched. Planning a refactor, tracing a bug across a serverless function and the client that calls it, reading a suite of Playwright specs to see what they really assert. It's the harness behind most of the larger changes on this site, and it's the one that respects a plan file well enough that I can approve the plan and walk away.
Codex does well in a scoped terminal loop: inspect a few files, patch, run the targeted test, summarize. When the task is "make this one behavior change and update its test," Codex tends to stay inside the lines.
GitHub Copilot is inline acceleration. Fixtures, repetitive edits, the third page object that looks like the first two. I don't plan with it. On good-vibes I gave it one job, reviewing commit diffs for style and consistency, and wrote that down in @@INLINECODE_0@@ so the role stuck.
Amazon Q Developer is what I use at work, in Visual Studio and VS Code, on a codebase that isn't mine to publish. The mode split between asking and acting is the part I lean on most, and I'll write more about that separately.
That good-vibes setup is worth spelling out because it's the clearest small example of the idea: three assistants, three written roles. Codex owned planning and the roadmap in AGENTS.md. Claude Code owned implementation and tests in CLAUDE.md. Copilot owned review. None of them had to guess what the others were for.
Where the model still matters
Once the harness is settled, the model is a capability and cost decision.
The chat on this site makes that concrete. Visitors see three tiers, Fast, Balanced, and Thoughtful. Behind those are three free-tier models on OpenRouter, and behind each of those is a local Ollama model I can run on my own machine with roughly the same character. The mapping lives in one place, functions/model-config.js, and the rule is that every production model has a local stand-in. If OpenRouter retires a model, and it has, the chain falls through to the next one and eventually to a managed free router. I'd rather answer a question with a smaller model than show an error.
That's the whole model-selection philosophy in one feature: pick the cheapest model that clears the quality bar for the task, keep a fallback, and make sure I can reproduce the behavior locally before I ship it.
Where local models fit now
Local models are useful for bounded work. Prompt drafting, which is what Promptpad does against Ollama. Generating test data without sending anything to a third party, which is what synthetic-data does from a browser tab. Enhancing a terse task description before it goes to a coding agent, which is what hopper does before it queues work for Claude Code or Codex.
I still don't hand a local model a long multi-file change and accept the result on faith. The verification loop has to be tighter, and the scope has to be narrower. But "interesting toy" stopped being the right description a while ago. Local Crew exists because I wanted to see how far a pool of ordinary machines on my own network could go, and the answer was further than I expected.
The metrics I keep
I stopped arguing about which model is best and started tracking things I can measure on my own work:
- Time from prompt to a passing test
- How many corrections I make before I'd merge it
- Regressions that show up after deploy
- Whether the handoff note told a reviewer what they needed
A setup that looks impressive and fails those is a demo. A setup that passes them is a tool.
The short version
Treat the harness and the model as two decisions. The harness sets how reliable the workflow is. The model sets what it can do and what it costs. Tests and scope control decide whether any of it is mergeable. That order has held up for me better than chasing whichever model shipped last week.