Phil · Claude · Kev · Cody
Two years of building, testing and revising a racing model. This is the latest chapter of an experiment about judging AI by its results.
Read the February–August field story ↗Recent race days
Selected examples, including the misses. The full ledger keeps the denominator visible.
Recent iterations · 2026
Feb–May 2026
The first spreadsheets looked the part. Claude then wrote persuasive cases for horses, adding challenge passes and live-day signals as mistakes accumulated. But the weights were still reasoned by hand. There was no fitted test of whether they meant anything.
August 2026
The first backtest appeared to find 61.7% of winners. Career summaries already included the race being scored. Once that leaked information was removed, the apparent magic disappeared. The hand-set model found 15.1% of winners against 14.5% for random picks.
August 2026
The fitted v5.1 model reported 26.4% top-pick accuracy and 71.5% winner-in-four coverage on its historical holdout. The market benchmark still did better. That August holdout is now a reproduction set, not fresh evidence for later variants. At Rosehill on 29 August, the model found useful shortlists but its probabilities scored slightly worse than equal chances.
September 2026
Base and day-adjusted selections are published together, so the live layer can be judged against the model before it. A good narrative about a hot jockey is not enough: it has to improve the results.
3 October 2026
At Randwick the lean won none of nine scored races. The winner was in the box seven times, and Pick 4 won four races. The experiment’s focus shifted towards the shortlist, while probability scores remained part of the evaluation.
17 October 2026
The current rule allows live-day signals to lower confidence, never raise it. The Caulfield Cup also exposed unreadable overseas form. Coverage flags are planned for r3.1 to make thin evidence visible; that does not turn missing evidence into a better prediction.
Historical account adapted from Claude’s 18 October retrospective. Earlier backtest figures are reported historical findings, not a new validation of the live model.
The hard lessons
The box caught 32 winners; the lean won 11. The shortlist and its ordering answer different questions.
Contested and being tested in public. Across these five meetings: three additional winners in the box, no net gain in winning leans. That is too small a sample to settle the question.
The historical distance component ran backwards. Repeated exposure can indicate a known quantity, not an advantage.
Not established. Keep log loss and Brier scores alongside the easier-to-read coverage measures.
The idea lab
The day layer may lower race confidence, never raise it.
Test whether more earlier results should be required before changing the four.
Test whether an early track pattern is fading.
A label that promised more certainty than the evidence supported.
Unknown official data stays missing and neutral.
Make unreadable form visible before relying on a selection.