Here is a finding that should change how you scope video projects. In a widely used multimodal video dataset, clips labelled as containing hateful content averaged 2.56 minutes in length — but the actually hateful segments averaged only 1.71 minutes. Roughly a third of the content inside those positively labelled videos did not match the label at all.
Nobody annotated anything incorrectly. Every clip was correctly identified. The label was simply attached at the wrong granularity, and the model inherited a third of its positive class as noise.
That is the real lesson of multimodal annotation. The modality does not just change the tool. It changes what the unit of annotation should be — and getting that unit wrong produces datasets that pass every quality check while teaching the model the wrong thing.
Text: the unit is the span, and negation is the trap
Text annotation looks the cheapest and produces the subtlest errors. The unit is usually a span — an entity, a relation, a sentiment-bearing clause — and the hard part is not identifying spans but handling linguistic structure that flips meaning.
Negation is the classic failure. In clinical and technical text, correctly identifying that a finding is absent is as important as identifying that it is present, and a span-level pipeline that ignores scope will treat both identically. Add coreference, hedging and nested entities and you have a task where guideline precision matters far more than annotator throughput.
What changes: guidelines carry the weight. Budget for several revision cycles before production.
Image: the unit is the region, and the taxonomy is the risk
Image annotation is the most mature workflow and the easiest to scope. The unit is a region — a box, a polygon, a mask, a keypoint set — and the main variables are label granularity and boundary precision.
The failure mode is taxonomic rather than technical. A class list that seemed clean in planning meets real images containing partially occluded, ambiguous or overlapping instances, and annotators start making individually reasonable decisions that are collectively inconsistent.
What changes: the taxonomy and the occlusion rules. Define what counts as a labelable instance before, not during.
Audio: the unit is the time span, and the boundary is subjective
Audio introduces a dimension text and images don’t have: annotators must decide when something starts and stops. Speaker turns overlap. Utterance boundaries are genuinely contestable in natural speech. Background events have soft onsets.
Transcription adds its own layer — orthographic conventions, disfluency handling, dialect representation — and inconsistency there propagates directly into speech model behaviour.
What changes: boundary tolerance has to be specified numerically, and agreement has to be measured with time-aware metrics rather than string match.
Video: the unit is the track, and identity is the metric
Video is not a collection of images. The dominant requirement is that an object keeps the same identity across frames. If a vehicle carries one track ID at frame 10 and a different ID at frame 20, the model learns fragmented object continuity rather than persistent objects — and per-frame accuracy will look excellent while the dataset is teaching the wrong lesson.
Three conventions have to be decided explicitly, because each needs different rules and different QA: what happens during occlusion, what happens when an object leaves and re-enters frame, and how re-identification is handled. Teams that treat these as edge cases discover they are the majority of the interesting data.
Cost scales differently too. A minute of video at standard frame rates is well over a thousand frames, so dense labelling is resource-intensive even with interpolation and semi-automated tracking carrying part of the load.
What changes: everything. Treat video scoping as a separate exercise from image scoping.
Where multimodal projects actually break
Not inside any single modality — at the joins. A visual label can be locally correct and still wrong in context, because it contradicts the paired audio or text, or because it is temporally misaligned by a few hundred milliseconds. Consistency within one format is a QA problem. Consistency across formats is a coordination problem, and it needs synchronised tooling, shared versioning and a QA pass that explicitly checks cross-modal agreement.
The honest trade-off: this costs more and forecasts worse. Annotation time per sample rises and estimation confidence falls, because the work depends on interactions rather than item counts. Teams that budget multimodal work at the sum of its single-modality parts run over.
The scoping question to ask first
Before selecting a tool or a vendor, answer this: what is the smallest piece of data that carries a complete label? A document, a region, a track, a 200-millisecond window, an utterance. Everything downstream — guidelines, agreement metric, cost model, QA design — follows from that answer, and getting it wrong is what produced the 33% noise in the example above.
How Shaip Supports Multimodal Annotation Programmes
The failure this article describes is a scoping failure, not a labour one — the wrong annotation unit chosen before anyone starts work. Shaip’s data annotation practice runs modality-specific workflows rather than one generic pipeline: span and relation annotation for text with negation and coreference handling, region and keypoint work for images, time-aligned transcription and event tagging for audio, and persistent-identity video annotation with defined occlusion, exit and re-identification conventions. For multimodal datasets we run cross-modal QA explicitly, checking that labels agree across formats and align temporally rather than validating each stream in isolation. Guidelines are hardened in a pilot pass before production volume begins.
Scoping a multimodal dataset and unsure what the annotation unit should be? Talk to our team before the guidelines are written — it is much cheaper than fixing it after.
