DEB JET AI UAT
Seven links, run in order. The value of this pipeline is not the list of steps — it is that the order cannot be reversed.
DEB JET has no built-in AI that assesses brokers. So the “AI under test” here is an external LLM (Claude, ChatGPT) acting as a UAT evaluator: it reads the evidence captured from a broker's journey and produces an assessment.
Two consequences follow, and they govern the whole design. First: the AI knows only what we hand it, so packaging the evidence is the core of the work, not a side task. Second: the prompt is the measuring instrument — the same evidence under two different prompts yields two different assessments. The prompt is therefore fixed and stored alongside the result.
Seven links
Find out what principles the field actually uses to evaluate AI output. The point is not to collect frameworks, but to avoid inventing criteria from nothing.
Produces: a research summary
Filter that research down to the criteria that genuinely apply to DEB JET, then restate them in this system's own terms. Importing a framework wholesale produces criteria too generic to test against anything.
Produces: the criteria set, each with a stated method of measurement
Turn each criterion into a numbered line with an expected result written in advance, the evidence it requires, and explicit Pass / Review / Fail thresholds. The thresholds must be sharp enough that two people scoring independently reach the same verdict.
Produces: checklist AI-001 … AI-008, reusable across any loan
The line you do not cross
Step 3 must be finished before step 4. That means writing down what the AI has to say to count as correct before seeing what it says.
Do it the other way round — run first, read the AI's answer, then weigh whether it was right — and you will always find a justification for whatever came out. That is no longer evaluation; it is rationalisation.
Define the evaluation criteria before running the scenarios. Do not decide whether the AI is correct only after seeing its output.
Walk the real journey in the UAT environment and capture evidence: screenshots, time spent per step, validation-error counts, backward navigation counts. For the weak-broker scenario these signals are the foundation — if the evidence is only clean screenshots, the AI has no basis on which to conclude anything about skill.
Produces: an evidence package per scenario
Hand the evidence package to the AI with the fixed prompt, and record the output verbatim. This is where the first question in the brief is answered — what information did we give the AI? — so each input package is logged as sent, never edited afterwards.
Produces: the AI's verbatim output plus the prompt used
Put the actual output beside the expected result written at step 3, and score each checklist line. Because the expected result came first, scoring leaves no room for hindsight.
Produces: an Expected / Actual / Verdict table
State the gap between expected and actual, explain why it opened, and recommend a change. The gap may lie in the AI — or in evidence we supplied too thinly. Telling those two apart is what makes the report worth reading.
Produces: a gap and recommendations report
Each checklist line takes exactly one of three verdicts. The boundaries between them are defined up front, not negotiated while scoring.
The brief asked for a folder holding the scenarios and the feedback. Each group of links has its place in it.
| Folder | Links | Contents |
|---|---|---|
00-framework/ | 1 – 3 | Criteria, checklist, scoring rules, fixed prompts |
01-howto/ | — | Instructions for others to run it themselves |
02-scenarios/ | 4 – 5 | SC01 strong broker, SC02 weak broker, SC03 system feedback |
04-report/ | 6 – 7 | Comparison table and cross-loan analysis |
Running the same persona against both Refinance and Purchase is how the AI's consistency gets tested. But the two loans differ on two levels — as products and by persona — so a rule is needed to avoid the wrong verdict.
The conclusion about the broker must match; the details about the system are allowed to differ. If one run classifies the broker as strong and the other as weak, that is a Fail. If both say strong but surface different friction points, that is correct behaviour — the journeys really are different.
This also strengthens scenario 3: an issue the AI reports on both loans is a system-level problem and a strong P1 candidate; one appearing on a single loan belongs to that flow and ranks lower.
Round results — 17 Sep 2026
Seven scripted runs, five of them valid — three are withdrawn, created by clicking the wrong button and measuring /loan/ instead of /loanjet/. One AI assessment made in isolation, one audit of that assessment. Full detail in 04-report/comparison.md.
| Signal | strong-refi | weak-refi | strong-jetsmsf | weak-jetsmsf |
|---|---|---|---|---|
| Route | /loanjet/ | /loanjet/ | /loanjet/ | /loanjet/ |
| Loan state | populated | populated | blank | blank |
| Next on empty form | advances, no errors | advances, no errors | disabled | disabled |
| Field for recommendation reason | present, optional | present, optional | present, optional | present, optional |
| Step marker | 11/11 | 11/11 | 11/11 | 11/11 |
Behaviour tracks the state of the loan, not the persona and not the journey. Three populated loans let an empty form through; two newly created ones disable Next. All five runs are on /loanjet/, so the route cannot be the cause.
This conclusion replaces two earlier ones. First we thought weak brokers hit a wall strong ones avoid — wrong, because persona was confounded with route. Then we thought the difference tracked the journey — also wrong, because the two runs added to test that were on a different product. No defect in this package is attributable to broker skill level.
Rule: leaves the broker without a compliance record. Reproduces on both loans.
input-brokernote exists on all three /loanjet/ runs but required: false; it is absent on both SMSF runs. A broker can pass the product step having recorded no reason at all. Under ASIC RG 273, this is where that evidence would have to live.
Rule: reproduces on both loans.
On all three /loanjet/ runs — both loans, both personas — Next on an untouched product step moved the file forward with errorsShown: [].
Rule: blocks completion. Reproduces across both personas.
reasonShownOnScreen: null while the control is disabled. The five errors on screen name other fields entirely.
This was our most strongly evidenced finding — 55 of 55 steps — and it was wrong. The probe looked for the active class on the tab button; the marker lives on the .rc-steps-item ancestor. Re-testing against the live system showed the system marks the step correctly. Kept here rather than deleted, because how it survived every automated check is the lesson.
Four successive malformed entries into loan amount — abc!!, 500.000,00, $500k, five hundred thousand — produced an identical five-error list every time, none of which mentions loan amount.
aria-current is 0 on every step. The marker is visual only.
Pass in every run. Enter an amount, move forward two steps, come back — the value is intact.
The largest lesson of this round
The most serious error was ours, not the AI’s. The AI repeated P1-2 faithfully because the evidence package we handed it said so. No amount of auditing the AI would have caught it — only re-testing the claim against the live system did.
A tool that measures the wrong thing produces consistent, confident, wrong numbers — and consistency reads as proof.
In short
This pipeline exists to enforce a single discipline: write the expected result before you see the answer. Everything else is procedure in service of that.
What it buys is a UAT that is traceable, repeatable and evidence-based — rather than resting on the scorer's subjective impression.