Building an AI product that refuses to lie

Amazon reads a resume for its leadership principles. McKinsey reads for structured problem solving. A Series B startup reads for whether you have shipped anything alone. The same bullet is strong at one and invisible at the next.

So students paste the job description into ChatGPT and get back something that reads well and did not happen. Calibr researches the company live, rewrites each bullet in its language, and marks every change for the user to approve.

I started from the complaint, not the idea. Before writing any engine I tore down eight tools, Teal and Rezi and Jobscan and Kickresume among them, and read what their users complained about on Reddit, Product Hunt and G2.

• 25+User interviews, run against a fixed script with an insight tracker, so I could tell a pattern from a loud opinion.
• 8+Competing tools torn down feature by feature, including their pricing and their output quality.
• 100+Complaints mined from public reviews and forums, coded into themes.

They fell into two piles, and the split is the whole thesis. Keyword tools were safe and useless: they matched strings and said nothing. General LLMs were useful and unsafe: better writing, and invented metrics their users then had to defend in a room.

Plenty of resume problems were worth solving. I picked this one because the incumbents were failing structurally, not from lack of effort: keyword matchers cannot read a company, and general models cannot be trusted not to invent.

So I wrote down what good meant before the engine existed, because "the output looks better" is not a bar anyone can fail. Three, in priority order:

• Interview-safe. Every claim in the output must trace to something the user actually wrote in their input. Zero invented facts is a pass or fail, not a score.
• Company-specific. Run the same resume against two different companies and the outputs should be materially different, not the same text with swapped nouns.
• Faster than doing it by hand. If it does not beat two hours of manual tailoring, nobody comes back.

The order matters more than the list. Interview-safe sits above company-specific, so whenever the two conflicted I shipped the more boring output.

A research pass, then a constrained rewrite, with a hard guard between the model and the output: if a rewritten bullet contains a number, a tool or an outcome that is not in the user's original input, it does not ship. The user approves every change.

It cost three real things.

• The output is less impressive. Put Calibr next to ChatGPT on the same resume and ChatGPT often looks better, because it is allowed to write the metric the user wishes they had. I lose some first impressions to that.
• It is slower and costs more per run. Two model calls plus live company research is a heavier request than one prompt.
• It puts work back on the user. Reviewing every change is friction. Some people want the magic button.

I took the trade because the promise is the interview, not the document. A resume that gets you into a room you cannot defend yourself in is worse than no resume. The guardrails have only ever been tightened, including when loosening them would have made a demo go better.

For six weeks I judged quality by reading it. That is a mood, not a method, and it does not survive comparing two versions of a prompt a month apart.

So I wrote ten criteria covering company fit, evidence grounding and bullet structure, and had a model score every run against them. LLM-as-judge. A number I could compare across versions instead of an impression I had to trust.

• 10Criteria in the rubric, written before any scoring, so I could not tune the bar to the result.
• 100+Real outputs scored, not hand-picked examples.
• 67 → 93Aggregate quality, before and after I rewrote the optimization logic against what the scores exposed.

It immediately caught a failure I could not see: quantification scored 9.5 on every consulting role and 7.5 on every technical one, while the average looked healthy. By hand I would have kept missing it, because I was reading the ones that looked fine.

That is the argument for evaluation in a sentence. It is not there to prove you were right. It is there to find the segment you were not looking at.

I would build the eval in week one, not week six. Every prompt change before it existed is one whose effect I cannot tell you. The rubric took an afternoon. The weeks of unmeasurable iteration cost far more.

I would have watched cost per run from day one. Every cache miss ran company research twice, uncapped, for weeks. Capping it cut search volume per calibration 71%. Never a hard bug, just a metric I had not thought to instrument, which is a product failure rather than an engineering one.

I would have picked a north-star metric on purpose. Mine emerged from whatever I happened to be looking at, so activation and retention took turns being the thing I cared about. Picking one would have made a lot of arguments with myself shorter.