How to Build a Custom LLM Evaluation You Can Trust

In the previous post, I published my Opus 5 and Fable 5 results. This post covers the reusable part: how I built a custom LLM evaluation, designed the questions, and scored the answers.

The scores will age quickly. A new model can make them stale, while the test design is easier to reuse.

Why write your own tests?

Public leaderboards have two problems.

First, they do not measure your work. A leaderboard may test competition math, SWE-bench, or long-context recall. Your daily prompt might be “merge these messy CSV files into one table” or “find the bug in this code.” A benchmark score does not necessarily predict performance on either task.

Second, you cannot inspect the failure. A leaderboard reports 43.3% versus 33.7%, but you may not know what the questions looked like, what went wrong, or whether one narrow category caused the gap.

I designed and ran the eight tasks in less than an afternoon. One rule had to be right from the beginning.

The hard rule: do not score by feel

The evaluation is only as useful as its scoring. If I compare two answers by deciding which one “feels better,” I also measure my preferences.

I used exact checks whenever possible and blind review for the one task that needed judgment.

The eight tasks ended up with this split:

Seven of the eight tasks had mechanical scoring. I chose that split on purpose.

Six ways to design the tasks

1. For unique-answer problems, verify your own answer first

Math and logic problems are easy to write, and easy to write incorrectly. If the evaluator’s answer is wrong, the entire task is worthless.

I wrote an independent verification script for each problem. One logic task assigned five people to five services across Monday through Friday under nine constraints. I brute-forced every permutation:

solutions = []
for svc_perm in permutations(services):
    for day_perm in permutations(days):
        # Check all nine constraints one by one.
        solutions.append(...)
print("solutions found:", len(solutions))   # Must equal 1.

The script first proves that the answer is unique. If it finds two solutions, the prompt is underconstrained, and I might mark a correct model answer as wrong.

The same applies to math. I independently checked three small problems with enumeration or exact fractions: the number of unordered pairs where lcm(a,b)=N, the expected flips before three consecutive heads, and the number of strings with no AA and an even count of B. I did not trust mental arithmetic.

2. Use hidden tests for algorithm tasks

I asked each model to implement topo_min(graph), which returns the lexicographically smallest topological ordering. A hidden pytest suite judged the code. I did not read the implementation and guess:

def test_cycle_raises():
    with pytest.raises(ValueError) as ei:
        topo_min({"a": ["b"], "b": ["a"]})
    assert "cycle" in str(ei.value).lower()

def test_input_not_mutated():
    g = {"a": ["b"], "b": []}
    snapshot = {k: list(v) for k, v in g.items()}
    topo_min(g)
    assert g == snapshot

Nine cases covered the edges where implementations tend to diverge: an empty graph, a self-loop, duplicate edges, nodes that appear only as successors, and input mutation. The happy path did not separate the models in this test.

I also wrote a reference implementation and compared model outputs against it. The evaluator needs an executable correct answer. Otherwise, code review becomes educated guessing.

3. Seed bugs before asking a model to find them

For the bug-hunting task, I gave each model roughly two hundred lines of log-analysis code and asked for every functional bug. I had planted six in advance, so the score was simply how many it found.

I also varied the difficulty. The six bugs were:

1. rec["status"] > 500 should be >= 500, because the original misses status 500 itself.
2. sorted(counts, key=...) lacked reverse=True and returned the least frequent endpoint.
3. sums[d] // counts[d] used floor division and truncated the average.
4. if summary["error_rate"] is 0.0 used identity comparison on a float.
5. merged = a followed by update(b) mutated an input in place.
6. "latency_ms": int(latency_s) silently discarded records with decimal latency values.

Both models found the first five. Both missed the sixth, which was the most dangerous because of where it hid:

try:
    rec = {
        "latency_ms": int(latency_s),   # "12.5" raises ValueError here.
        ...
    }
except ValueError:
    continue                            # The record then disappears.

The bug is difficult to spot because it sits inside ordinary-looking error handling. try/except ValueError: continue is a common defensive pattern, easy to skim past. Here, the “defense” created the path to data loss.

This was my favorite result in the entire evaluation. The two models caught every obvious bug and missed the same one disguised as defensive code.

4. Turn instruction following into hard constraints

I asked the models to write a product announcement under seven constraints: exactly eight lines, a numbered prefix on every line, 10–15 words per line, every line ending in shipped., no letter z anywhere, dark mode appearing on exactly two lines, and no commas.

The scorer was only a few dozen lines, with one Boolean per constraint:

results["C3_10_to_15_words"] = all(10 <= len(l.split()) <= 15 for l in lines)
results["C5_no_letter_z"] = "z" not in text.lower()
results["C6_dark_mode_twice"] = sum("dark mode" in l for l in lines) == 2
results["C7_no_commas"] = "," not in text

Constraints should interfere with one another. Writing 10–15 words per line, avoiding commas, and ending on a fixed word creates pressure across the whole answer. Any one rule is easy. Satisfying all seven at once is what makes the task useful.

5. Hide agentic-task traps in the data

For an agentic task, the final number is not enough. The processing path matters. I supplied three order files with different schemas and asked the model to merge them:

orders_east.csv    order_id,customer,amount_usd,date,status
                   1001,Acme Corp,250.00,2026-06-01,completed

orders_west.csv    id,client_name,total_cents,order_date,state
                   2001,  Acme Corp ,50000,06/06/2026,completed

legacy_orders.csv  OrderID;Customer;Amount;Date
                   3001;Hotel BV;42.00;10-06-2026

The data contained six failure points: a semicolon delimiter, an amount in cents, three date formats (ISO, month/day/year, and day-month-year), whitespace around customer names, duplicate order IDs requiring source priority, and cancelled orders that had to be removed.

The dates were the nastiest part. 06/06/2026 and 10-06-2026 are both valid, but one puts the month first and the other puts the day first. A single row does not reveal the convention. The model has to infer it from other rows in the same file.

The validation script compared every output row and checked three aggregates:

checks["merged_rows_exact"] = rows == EXPECTED_ROWS
checks["total_revenue"] = abs(tr - 1572.09) < 0.005
checks["top_customer"] = summary.get("top_customer") == "Acme Corp"

Total revenue is a compact checksum. Keep one cancelled order, mishandle one unit, or deduplicate the wrong record, and the number no longer matches. One total detects all six classes of error.

6. Blind-review subjective work and reverse the order

Only one task resisted mechanical scoring: an architecture decision memo to a CTO from the perspective of a platform-engineering lead.

That task needed human judgment or another model, so I used three safeguards.

Author bias. The judges did not know which model wrote either document. The inputs were simply “Memo A” and “Memo B.”

Position bias. This is easy to overlook. One judge received A then B, and the other received B then A. A consistent winner after the swap is more likely to have won on content rather than position.

Rubric drift. I gave both judges an explicit scoring table instead of asking for a general impression:

Clear decision and persuasive rationale            0–3
Use of the supplied data and technical judgment    0–3
Risk identification and mitigation                 0–2
Feasibility of the 90-day plan                      0–1
Formatting and constraint compliance               0–1

I computed objective values, such as word count, before review and supplied them to the judges. At least for this kind of task, I do not treat a model’s estimate of its own word count as reliable.

Where my evaluation failed

The method also exposed mistakes in my setup.

1. Seven ties meant my questions were too easy

Both models scored 44 out of 45 on the objective tasks, and seven of the eight tasks ended in exact ties.

The scores could suggest equal ability. A better explanation is that my tasks were too easy to separate them.

This is a ceiling effect. A test where everyone scores full marks measures almost nothing. Useful questions live where one system succeeds and another fails. Only one of mine reached that region, and both models failed it.

Next time I will raise the difficulty or include a calibration set of tasks known to produce failures, so I can see where the ceiling begins.

2. The judges had variance too

The blind review produced an accident: the same judge ran twice and returned two different margins, 9.5 to 8.5 and 8.5 to 8.25.

The inconsistency was useful because it exposed model-judge variance directly:

• Absolute scores were unstable. With the same material and the same judge, the margin moved from 1.0 to 0.25.
• The ranking was stable. All three scorecards, including the other judge’s, put the same memo first.

I therefore used the ranking as a directional signal and ignored the absolute numbers. I would describe this result as “A was better than B,” not “A is a 9.5.” The same judge changed that 9.5 on its next run.

This is worth testing in other LLM-as-judge settings, and I plan to examine it separately.

3. “No tools” relied on trust

Several prompts prohibited tools and asked for pure reasoning. I had no way to enforce the ban. I could only state it and trust the model to comply.

Both sides ran under the same conditions, so the comparison was still reasonably fair, but the uncertainty remains. A real no-tools test must remove tool access at the execution-framework level. A prompt alone cannot guarantee it.

4. The agent framework added a confounder

Several prompts said “output only the answer.” Both models sometimes added a preface. I initially counted those as format violations, then noticed that the subagent framework itself required agents to return a summary.

The models may have been following the framework rather than disobeying my prompt. That contaminated the metric, so I kept it as an observation but removed it from the score.

When designing a test, inspect the execution environment for hidden instructions added to your prompt.

5. One run cannot measure variance

I ran each task once. Strictly speaking, the evaluation measured performance on those runs, not stable capability.

The judge returned two different scorecards from identical material, so one run can be noisy. Before making claims about reliability, I would run each task at least five times and inspect the distribution. This is the evaluation’s largest limitation.

A checklist you can reuse

If you want to compare two models on your own work:

1. Decide exactly what you want to measure. Start with abilities you use every day, not leaderboard categories.
2. Prefer mechanically scored tasks. Aim for at least 70% of the evaluation to require no human judgment.
3. Verify every answer first. Prove unique answers are unique, and keep a reference implementation for algorithm tasks.
4. Hide the scoring criteria from the models. Keep test cases and seeded-bug lists separate.
5. Vary the difficulty. Tasks that everyone passes or everyone fails have little discriminating power.
6. Blind subjective reviews and reverse the answer order, or you may be measuring position bias.
7. Copy the prompt exactly. Any wording difference becomes another variable.
8. Record process metrics. Runtime and tool-call count can reveal working style even when scores tie.
9. State the limitations honestly: the sample size, uncontrolled variables, and contaminated metrics.

The final totals were 52.63 and 53.25, a gap too small to mean much. I learned which tasks both models handled reliably, where they failed together, and how their working styles differed. I can reuse the same evaluation the next time I compare two models.