Methodology
Rigour you can take to a regulator.
Every clinical AI evaluation we run is calibrated against expert consensus, scored with proper rules, and reported with confidence intervals — reproducible from raw judgement to the final Reliability Report. The same standard builds the datasets we curate.
Two-phase calibration.
Every evaluator is calibrated against expert consensus before they assess your AI — first a fast screen, then a full statistical calibration. The same pathway runs for every profession, so a score means the same thing whoever earned it.
Phase 1 — Quick Screen
An 11-task sequential screen on gold-standard items with known expert consensus. Anti-luck guardrails and per-item agreement thresholds confirm an evaluator's judgement is aligned with the reference standard before they commit to the full calibration. ~20 minutes.
Result
Stopped at task 23 — CI narrowed below 5%
Score
91% ± 3.2%
Phase 2 — Full Calibration
A complete statistical calibration across all eight RLHF task types — rating, comparison, ranking, rubric, correction, annotation, justification, and red-team. Produces the reliability score with Beta-Binomial / bootstrap intervals per metric, and forms the basis of the Reliability Report. 60–90 minutes.
Proper scoring rules.
We score with mathematically proper rules that reward honest, well-calibrated judgement — and report the uncertainty around every number rather than a bare point estimate.
Brier Score
Measures the accuracy of probabilistic predictions. A proper scoring rule that rewards well-calibrated confidence — penalising overconfidence and underconfidence equally.
99% confidence on wrong answer → 10× penalty vs Brier
Log Score
Sharper discrimination between good and poor calibration than the Brier score — particularly good at catching evaluators who are systematically overconfident in their clinical assessments.
C-02: 86% point estimate but LCB 80% — below threshold
Confidence Intervals
Beta-Binomial intervals for proportions, bootstrap intervals for continuous metrics, and lower confidence bounds for certification — so every score reflects worst-case plausible performance, not just a point estimate.
Inter-annotator agreement.
We measure how much evaluators agree — correcting for chance — so we can tell genuine clinical disagreement apart from noise, and report the reliability of the process itself.
Observed agreement
75% (6/8 items)
Cohen's κ
0.71 — Substantial
Cohen's Kappa (κ)
Pairwise agreement between two evaluators, corrected for chance. Used for direct comparison tasks and pairwise annotation where each item is assessed by exactly two people.
Categories
Fleiss' κ
0.62 — Substantial
Fleiss' Kappa (κ)
Extends agreement to multiple evaluators — essential for production work where items are assessed by varying numbers of clinicians and we need the overall reliability of the process.
Five layers in production.
Calibration gets an evaluator in the door. These five controls keep the quality there — running continuously through live work, not just at the start.
Gold-standard injection
Known-answer tasks are randomly seeded into production queues. Evaluators can't tell which items are gold, so quality holds on every response — not just when they think they're being watched.
Drift detection
Statistical monitoring flags when an evaluator's performance starts to drift from their calibration baseline — catching fatigue, disengagement, or shifting standards early.
Attention checks
Strategically placed verification items confirm evaluators are reading and reasoning, not pattern-matching or rushing.
Skip budget
Evaluators can skip tasks outside their expertise, within a managed budget — preventing gaming while allowing honest acknowledgement of limits.
Justification quality
Multi-signal analysis of written rationale assesses depth, clinical reasoning, and consistency — surfacing thin or formulaic explanations that signal disengagement.
Readiness scoring.
Before any paid work, we compute a readiness score across three task families — so the right evaluator is matched to the right project, with the evidence to back it.
Ranking & Correction
Pairwise ranking, text correction, and response assessment — the core RLHF task families.
Safety & Red-Teaming
Adversarial testing across failure-mode categories — identifying dangerous outputs before they reach patients.
Data & Annotation
Clinical data labelling, document classification, and structured annotation for AI training datasets.
High-quality datasets, built sovereign.
The same rigour that grades a model can build the data that trains it. We apply our sovereign methodology to commercial datasets too — the same standard, kept firewalled from the sovereign programme so your data and IP stay yours.
Clinically authentic
Grounded in how senior UK clinicians actually reason about real patients — so the data carries the judgement that makes medical AI safe where it matters.
Built to last
Clinical practice never stands still, and neither do our datasets. They're built to stay anchored to current UK practice, so quality holds up long after delivery.
Authored, not scraped
The expertise that matters can't be scraped from the open web — it comes from verified UK clinicians, captured at the depth and rigour serious medical AI demands.
Same methodology. Firewalled from the sovereign side.
Your commercial datasets are built to the same standard — but kept strictly firewalled from our sovereign programme. Your data and IP stay yours: curated and held onshore under a documented, auditable process, never folded into the shared sovereign asset. The methodology behind it is our core IP, shared only under NDA.
Every engagement ends in a Reliability Report.
A procurement-grade artifact — reproducible on demand, with the full methodology referenced throughout. Built for the people who ask the hardest questions: procurement, regulators, and your safety review board.
- 01
PDF + JSON
Human-readable for safety reviews; machine-readable for ingestion into your QMS or evidence pipeline.
- 02
Per-metric confidence intervals
Beta-Binomial intervals for proportions, bootstrap for continuous scores — a lower confidence bound on every metric.
- 03
Readiness scores
Overall, safety-critical, and annotation-heavy readiness, each backed by the underlying calibration data.
- 04
Full audit trail
Evaluator IDs, gold-task agreement, attention-check history, and methodology version recorded against every result.
30-minute call. Get a quote.
Tell us what you’re building and what you need evaluated, red-teamed, annotated, or generated. We’ll come back with a fixed-price brief — no long enterprise procurement.
- Fixed-price quote
- No drawn-out procurement process
NDA available on request.
entertheloopClinicians powering AI alignment, training & safety.Verified against
entertheloopClinicians powering AI alignment, training & safety.