Nineteen of Twenty

On 15 September I deleted fifteen cases from SPURIA. They weren’t drafts. Every one was built, sourced, seeded into the database, played through and priced. Together they were most of a series called Loose Files, and it had taken me two days to build it.

A model with web search had solved nineteen of their twenty fields. Nothing in the questions was wrong. The answers were already written down somewhere, so the cases had no work left in them for an analyst to do.

What the Loose Files were

SPURIA has two kinds of case. Arc cases form a chain: each one hands the next an opening lead. Loose Files stand outside that chain. You can open them in any order, but they share the same ground (Hong Kong), and they carry the hard parts: a fixed attempt budget, higher difficulty, more points.

The first batch, SPU-L-006 to SPU-L-020, went up on 13 and 14 September, across three disciplines. Most of them were register and knowledge work. Who controls this company now? When did this network first reach the island? Which denomination does the government issue itself? Each question came with a researched, sourced answer and a careful list of near misses that the grader would reject. It was honest research, and it was also the problem.

The copy-paste moment

It started with one field. SPU-L-012, Finding 02, asked:

The Hong Kong Observatory runs a station on that summit. At what height in metres does the Observatory itself list the station?

I copied the prompt, pasted it into a search engine, and the AI overview at the top answered 955 metres, with a source card next to it. That took one line, and it was the whole second chain of the case.

The field wasn’t a one-off mistake. It named its own object (the station on that summit) and its own source (the Observatory itself). Whoever writes both has already written the search query, and all the analyst has to do is press enter.

So I tested every finding in the catalogue the same way, forty-one in total. Does the prompt name both its object and its source? Does the answer sit in a sentence on the open web? Does reaching it need a tool?

The breaking fields had a clear pattern. They were knowledge questions: correctly researched, properly sourced, and done in a single line now that search engines answer for you.

Over the next day I rebuilt them. I took the object out of each prompt and made it come from the material or from the previous finding. When I did that, I noticed something else: the briefing kept pulling back in the exact word the prompt had just lost. A field is only rebuilt when claim, briefing, prompt and hints have been checked together. By the evening of the 15th, no prompt in the catalogue contained its own search query any more.

The stronger test

I ran a second test the same day. The first test had asked whether the answer shows up when you paste the prompt. The second asked something harder: what happens if you hand the entire case text to a model with web search?

Nineteen of twenty fields fell.

Once you run that test, the wording of the prompt stops mattering. The model reads the claim too, writes its own search query, and opens the PDF sitting behind the query form. Three results from that run contradicted things my own test document had claimed a day earlier:

The rebuilt fields didn’t hold either. For one case the model reproduced, word for word, the derivation I had just moved out of the prompt and into the evidence document. For another, it built its search query from the claim (“1979, Central, underground”) and got both findings at once. The prompt I had rewritten played no part at all.

This is the point where rewording ran out. A query form only protects its data as long as that data doesn’t also exist somewhere else as a document. Arbitration awards get published as PDFs, company registers as memoranda, and reportable incidents as official notices. If the answer is a published fact, a model will find it, however the question is phrased. The flaw was in the answer itself, and the answer is the one part of a case you can’t rephrase.

So I deleted them: fifteen cases, from the dev database and from the casebook. The seeds are still in git history, but they aren’t coming back as cases.

What held

Exactly two kinds of field held.

A computed value. Also in SPU-L-012, the second finding asked for a station identifier that isn’t written in any sentence anywhere. It only appears after you download a year’s file from an open measurement series (2,922 rows) and sort by the right column. The model got it wrong: asked the same question, it gave a long-term average where the question wanted a single reading. It was the only field in the run that didn’t fall.

The image cases. The arc cases survived intact, because their objects are inside photographs. I checked every alt text: not one contains any of its own case’s answers. Pasting the page text doesn’t carry the object along with it. This property has to be built, and checked again with every new image case, because a more descriptive alt text would switch the protection off.

Out of that came the rule that now sits in the casebook’s construction grammar:

A field whose answer exists anywhere as a sentence is an easy field. Medium and above require at least one field whose answer exists nowhere as a sentence — because it only emerges from a calculation over a dataset, or because it can only be read in the material.

The second attempt, and the second deletion

On 16 September I built fifteen new cases, SPU-L-022 to SPU-L-036, on exactly that rule. Every one pointed at a live open dataset: immigration passenger counts, fare tables, interbank liquidity, certificate transparency logs, tide heights, bus routes. Every answer had to be computed and appeared in no sentence anywhere.

This time I ran the model test before calling anything finished. It covered thirty fields.

Not one fell to search. The rule had worked. Twenty-nine fell anyway, because a model with network access downloads the dataset and runs the aggregation itself. It did this faster than a person would, without dead ends and without needing a hint. I had never claimed to prevent that. My mistake was building cases where running the aggregation was the only thing to do.

The test models said it themselves: “the briefings name the exact file, and each question maps to one groupby”, “a data-entry exercise, not a puzzle.” In one case, a single URL that the briefing nearly spelled out returned both answers in one line of JSON.

These fifteen weren’t broken the way the first batch was. They were pre-chewed: the flaw sat in the briefing, which is text, and text can be fixed. I deleted them anyway, because fifteen cases cut from one template, thirteen of which do the same thing, are not a base for a series. There was also a bonus: the test surfaced four factual errors nobody would otherwise have seen, including a wrong answer key and a description of a column that had it backwards.

Four rules came out of that day:

1. The briefing gives access, not procedure. Naming the publisher is fine. Naming the file, column, filter, parameter or date range is not, because whoever names them has done the investigation, and what’s left is a spreadsheet with instructions attached.
2. Check the key against the full dataset, not against your download. One assertion passed cleanly against 1,000 records from 2022 onward. The full series had 6,328 records going back to 2002, and 66 matching values instead of two. Anyone who followed the briefing correctly would have got a different answer from mine.
3. Check every hint against the live data. Several hints warned about traps that didn’t exist in the data. That costs the analyst points for nothing, and the warning gives away how the solution is structured.
4. If the material carries the answer, every field has to need it. The only field that held was one where twenty-four values had to be read off a chart and matched against the files. The attacking model’s own verdict: “genuinely unanswerable without the image.”

Labels in the chart

The next case was built on its own, and its material was two data charts matched against open datasets. Given only the text, a search-enabled AI mode found neither finding and asked for the charts.

So I gave an agent the charts, and it solved both fields in twelve tool calls.

The charts printed every value, so there was nothing to estimate, and the agent said as much. The first field became a lookup: compare the printed numbers against the dataset until exactly one row matches. The second took real digging, but digging is what an agent with tools is good at. Its own caveat was the most useful line in the report: without the labels, parts of the second chart couldn’t have been told apart at all.

The labels make the case solvable and they also make it trivial. That single design decision produces both results. For a person with a browser and a spreadsheet it’s a fair medium, about 25 to 40 minutes. Against a model with tools it offers no resistance.

Which kind of answer holds

Against a model with web search, here is what I’ve seen hold and what I’ve seen fail:

• Fails: any answer that exists somewhere as a sentence. That includes one behind a query form, since the data probably also sits in a PDF. It also includes one that needs “three tools”, when the result of those tools has been written down somewhere.
• Fails: a computed answer where the briefing describes the calculation. At that point the case is a groupby, and models are good at groupbys.
• Holds, for now: an answer that has to be calculated from a dataset, where the case doesn’t say which one or how, and where the widely repeated figure is wrong. An AI overview will confidently give the popular figure, and that works as a trap instead of a shortcut.
• Holds: an answer that can only be read in the material, from a photograph or a chart, where the page text (alt text included) carries none of it.

None of this is permanent. What needs a query today might appear in a blog post next year, and an overview will read that post. And none of it holds against an agent that has the material and the tools. That’s the next test, and the chart case failed it.

The rules I build by now are simple. I build one case at a time, with no batch production. Nothing ships until it has had two runs, a model probe and a full author run, both on the finished version. The difficulty I print on a case describes the work that remains once a model has had a go, not the work I did to build it.

The first batch was fifteen cases, and one field held. That one field is the pattern every case since has been built on.