Our Closing Line Value Measurement Was Broken for Two Months
We track the price we published against the price the market closed at. In July we found out we had been measuring it wrong since May. Three bugs, what they cost, and what the verified numbers look like now.
Every pick we publish gets a result: win, loss, void. That is the number everyone looks at, and in the short run it is the number that tells you least. To separate a real edge from luck using results alone you need several hundred settled bets, and by then the season has moved on.
Closing line value is the faster signal. If we publish a pick at 2.10 and the market closes that selection at 1.90, we were early on information the market later agreed with. The bet can still lose. Being right about the price and wrong about the result is normal, and over time the price is the better indicator. If you are new to the idea, we explain it in our guide to value betting.
We have been tracking closing line value since May. In July we found out we had been measuring it wrong the entire time.
The signal that something was wrong
At 133 measured picks, closing line value pointed in the opposite direction to profit. Our main picks showed an average CLV of minus 3.83% while returning plus 4.9% ROI. Our value picks showed plus 1.53% CLV while returning minus 22.7% ROI. Both backwards.
Closing line value can disagree with the result of a single bet. That is the entire point of it. It should not disagree in a consistent direction across two separate groups of picks. When your quality measure and your outcome measure point at each other, the honest first assumption is not that you have discovered something about the market. It is that your instrument is broken.
Bug one: comparing two different bookmakers
Our published odds come from one preferred bookmaker, chosen because a reader has to be able to actually take the price we show. The closing snapshot did not respect that. It took whichever bookmaker the odds feed happened to return first.
The spread between bookmakers on the same selection reaches 55%. On one fixture in our data the away price was 4.50 at one book and 7.00 at another, at the same moment, on the same market. Comparing a morning price at one book to a closing price at another does not measure line movement. It measures the gap between two companies’ risk appetite.
Bug two: a notation mismatch that deleted a whole market
Our odds provider writes Asian Handicap lines without a trailing zero: Home minus 1, Home plus 0. We wrote them with one: minus 1.0, plus 0.0. Every whole line and every zero line therefore failed to match. Not approximately. Never.
Asian Handicap made up 98 of 184 picks in the affected period, and those on whole lines dropped out of the measurement without producing a single error in the log. A silent filter is worse than a crash, because a crash gets fixed the day it happens.
Bug three: a substring comparison matching the wrong line
The third bug was the subtlest and the most damaging. Selections were compared with a substring check rather than a numeric one, so a pick on Asian Handicap plus 1.5 could match a completely different line that happened to share the same characters.
Five of fourteen Asian Handicap measurements had a closing price between 7.0 and 10.0 recorded against a taken price of 1.30 to 1.35. That is not line movement, that is a different bet. Those five rows alone dragged the average measured CLV from plus 0.41% to minus 2.40%, which is to say they were single-handedly producing the conclusion that our prices were consistently worse than closing.
What the coverage looks like now
Every other market combined went from 123 of 322 measured (38.2%) to 24 of 25 (96%).
What is still missing, and why
Asian Handicap sits at 69.2% rather than 96%, and the remaining gap is not a bug. Bookmakers do not quote every line on every match. On one fixture the available range ran from minus 2.75 to plus 0.75, so a pick on plus 1.75 has no closing price to compare against, no matter how correct the matching logic is. That is the market, and we would rather report a market at 69% coverage and say so than blend it into a single headline number.
Two other limits are worth stating plainly. Of 171 closing snapshots in the database, 149 belong to picks recorded before we started storing which bookmaker supplied the published price, so those rows cannot be used for a bookmaker-consistent comparison at all. Of the 22 that remain, 10 fell back to a different bookmaker at closing because the original no longer offered the market. And we have not published a 1X2 pick since the middle of July, so for that market there is currently nothing to measure either way.
What we are not going to claim
That leaves 12 picks measured correctly, at a consistent bookmaker, between 21 and 26 July. Their average closing line value is plus 0.66%.
Twelve picks. Three of them won. Together they lost 7.94 units.
We are not going to present plus 0.66% as evidence that our model beats the market, and we are not going to present minus 7.94 units as evidence that it does not. Neither number means anything at this sample size, in either direction. What we have is a measurement we now trust, and not yet enough of it. We will report when there is something to report.
The part that generalises
Two of these three bugs were introduced by fixes to the same system. The bookmaker fix was applied in the selection stage and in the save stage, but a converter in between silently dropped the field, so for a while the code looked correct in both places anyone would think to check while doing nothing at all. A half-applied fix is more dangerous than no fix, because it stops you looking.
What caught all three was not code review. It was noticing that a quality measure and a profit measure had been disagreeing for weeks and refusing to explain it away. The instrument is part of the product. If we publish our record and our ROI, we owe you the state of the ruler we measured them with.
