Seven experts segmented lesions on the same multiple sclerosis brain dataset. Their agreement with the consensus ranged from 0.66 to 0.76 median Dice score. Every one of them was a qualified specialist. Every one of them drew a different boundary.
This is the central fact of medical data annotation, and it is the one most vendor conversations skip. Hiring clinicians does not produce clean ground truth. It produces expert disagreement — which is more informative than crowd disagreement, but still disagreement. The programmes that succeed are the ones that measure it, fuse it deliberately, and document the whole process while it happens rather than reconstructing it for a regulator afterwards.
Why medical annotation is structurally different
In most annotation domains, a degree of annotator noise is tolerable — models are robust enough to learn through it. Medical AI has less margin, because the error does not average out into a slightly worse F1 score. It becomes a missed finding in a specific anatomical region.
The difficulty is not annotator quality. It is that the underlying object is genuinely ambiguous. Anatomical boundaries are ill-defined. Lesion margins fade rather than stop. A 2024 review in European Radiology Experimental set out the properties training data needs — consistent, standardised, traceable, correctly annotated and de-identified — and every one of them is a property of the workflow, not of the images.
Measuring agreement properly
Pick the metric that matches the task:
| Task type | Metric | What it tells you |
|---|---|---|
| Categorical labels, two readers | Cohen’s kappa | Agreement corrected for chance |
| Categorical labels, three or more | Fleiss’ kappa | Group-level reliability |
| Segmentation | Dice coefficient | Spatial overlap of masks |
| Detection / localisation | IoU | Boundary precision |
Published methodology for assessing agreement in medical segmentation combines quantitative metrics with agreement heatmaps that show where readers diverge, which is the more actionable output. A kappa score tells you there is a problem. A heatmap tells you it is concentrated in one lesion category, which is a guideline fix rather than a staffing one.
Constructing ground truth rather than collecting it

Once you accept that readers disagree, the question becomes how to combine them. Majority vote discards information. Single-reader ground truth bakes one clinician’s interpretive habits into the model permanently.
The established alternative is statistical fusion: methods such as Simultaneous Truth and Performance Level Estimation estimate a probabilistic consensus while simultaneously scoring each reader’s performance, and are used specifically to generate ground truth for training from multiple expert segmentations. The practical benefit is twofold — a better reference standard, and a per-reader reliability signal that tells you who needs recalibration.
A related point that costs teams dearly: inter-reader variability affects evaluation as much as training. Research on chest radiograph models identifies variability in ground-truth annotations as a limitation on both learning and assessment. If your held-out test set was labelled by a single reader, your reported sensitivity is a measurement of agreement with that person.
The compliance layer runs in parallel, not after
Three things have to be true while the annotation happens, not retrofitted:

De-identification. Protected health information has to be removed from pixel data and metadata both — burned-in identifiers in DICOM headers and image overlays are the classic miss. Under HIPAA this means a defined protocol, not an ad-hoc pass.
Annotator qualification records. For a regulated submission, you need to show who annotated what and what their clinical credentials were. A vendor who cannot produce qualification records on request does not have the clinical annotators being described.
Traceability. Dataset version control, audit trails and documented guideline revisions. When guidelines change mid-project — and they always do — you need to know which items were labelled under which version.
What we see go wrong
In our delivery work, the single most common failure is not annotator skill. It is a protocol that never defines the edge cases, so five clinicians resolve the same ambiguity five defensible ways and the disagreement looks like noise when it is actually an unwritten specification. The fix is a pilot pass on a small sample, a structured adjudication session on every divergence, and a guideline rewrite before production volume begins.
The complaint that comes up constantly among practitioners is the mirror image: the annotation spec was written by someone who has never read a scan, so it does not survive contact with a real edge case.
How Shaip Supports Healthcare AI Teams
The problem this article describes — turning irreducible expert disagreement into a defensible reference standard — is what Shaip’s medical data annotation programmes are built around. Annotation is performed by licensed clinicians matched to the modality: radiologists for CT, MRI, mammography and X-ray, pathologists for whole-slide images, ophthalmologists for fundus and OCT. Every project runs multi-reader agreement measurement, structured adjudication on divergent cases, and consensus construction rather than single-reader labelling, with DICOM-native workflows, documented de-identification, annotator qualification records and version-controlled guidelines. Where you need volume before you need custom collection, our medical data catalog offers licensed, de-identified imaging and clinical datasets. Compliance posture covers HIPAA, GDPR, SOC 2 Type II and ISO 27001.
Building toward a regulatory submission and unsure whether your annotation trail will hold up? Talk to our team before you scale the labelling, not after.