There is nothing in the product to point an evaluation at, so “test the AI” cannot mean testing a feature. We read it the only way that makes sense: the AI is the evaluator, not the thing evaluated — an external model looks at DEB JET and reports what it sees.

That reading creates a second problem immediately. Slide 03 is about it.

What we are delivering is not a report. It is a pipeline that produces reports — for this loan, and for any loan after it.

That distinction matters commercially. A report answers one question once. This answers the same question every time a new journey ships, run by someone who was never in this room.

If the AI is our instrument, then the AI can be wrong — and a wrong instrument produces a confident report full of things that never happened. So we split the evaluation in two.

Before we wrote a single criterion, we surveyed the system and listed what could actually be captured.

Criteria are written before anything is run. If you look at results first and then decide what counts as a problem, you are not evaluating — you are justifying.

Note where step 5 sits. The AI is asked fourth from the end, not first. By the time it sees anything, the criteria are frozen and the evidence is collected.

If you hand the AI your criteria and then check whether its output matches your criteria, you have tested nothing — you have tested whether the AI can repeat you. It can. Every model can.

Isolation is what makes the audit worth anything. When the AI independently finds something we found, that is corroboration. When it finds something we missed, that is value. When it claims something the evidence does not support, we catch it — that is quality control. All three happened.

Condition 3 is not a usability concern. Under ASIC RG 273, the evidence of best-interests compliance comes predominantly from the broker’s own records. If the journey lets a broker finish without recording a reason, they cannot produce that evidence later. The consequence is not a bad experience — it is an unmet regulatory obligation that surfaces long after the loan settled.

Condition 4 separates a system defect from a per-loan configuration issue. Something reproducing on Refinance and Purchase is a defect in the product. That distinction is what makes a P1 claim defensible instead of assertive.

Round 1 ran the weak broker only on one route and the strong broker only on the other. We then concluded that weak brokers hit a wall strong brokers avoid.

That conclusion was wrong, and the way it was wrong is instructive. Persona was confounded with route. When two variables move together, you cannot attribute the effect to either one. We had evidence of a difference and no evidence at all about its cause.

This cell had to be closed twice. Round 2 closed it by crossing two different routes — still confounded, just less visibly. Round 3 closed it properly: same loan, same route, persona the only thing that moved. See slide 11.

Seven scripted runs, five of them valid. Three are withdrawn: they were created by clicking the wrong button and measured /loan/, the SMSF product, rather than the /loanjet/ journey the brief asked about. Slide 11 has the detail.

Those three are not deleted. The AI's round-1 assessment genuinely read one of them, so removing them would falsify the audit trail.

The packager is deterministic: same inputs give a byte-identical package. That is what makes two runs comparable rather than merely similar.

The packager is deterministic: same inputs give a byte-identical package. That is what makes two runs comparable rather than merely similar.

Note the two loan states have opposite defects. A populated loan lets an empty form through; a newly created one blocks and will not say why. Both are on /loanjet/. Neither state tells the broker what is actually required.

The AI — looking only at the evidence — flagged the confound in our own scenario design. It was right. We ran the two missing cells and concluded that every difference tracks the journey.

That conclusion was also wrong. The two cells we added ran on /loan/ — the SMSF product's route, not /loanjet/. We closed the gap by introducing the very variable we were trying to eliminate. What we called a journey difference was a difference between two products.

Round 3 re-ran both personas on the same loan, on the same route. The table above is the result, and it gives a different answer:

Behaviour tracks the state of the loan, not the persona and not the journey. Three populated loans let an empty form through; two newly created ones disable Next. All five are on /loanjet/, so the route cannot be the cause.

No defect in this package is attributable to broker skill level. The matched pairs are identical on every captured signal.

The AI repeated the claim faithfully because our evidence package said so. It did exactly what it should have done. No amount of auditing the AI would have caught this — the AI was not the source of the error. We were.

What caught it was re-testing against the live system. We wrote verify-active.mjs, pointed it at DEB JET, and found the marker present and tracking correctly. We fixed the probe and repeated two runs; both report activeMarked: true on 11 of 11. A2-1 is a Pass.

One real finding survives the wreckage, now measured: aria-current is 0 on every step — the marker is visual only and is not exposed to assistive technology. That is genuine, it is P3, and it is nothing like what P1-2 claimed.

Recorded so that no silence is read as a pass.