Most teams training or fine-tuning a language model hit the same wall at the same point. The open corpora are exhausted. The scraped material is legally ambiguous. And the thing the model is worst at — Gulf dialect, code review on a real repository, clinical reasoning, screenplay structure — is exactly what no public dataset covers properly.
That gap is why we have expanded Shaip’s text data catalog. Eight categories of licensed text datasets for AI training are now available, each one scoped, documented and ready to license rather than assembled on request.
What’s in each category
The catalog at a glance:
| Category | Headline scale | Languages / formats |
|---|---|---|
| Conversational Q&A | ~172,000 human-authored Q&A pairs across 8 sets | Arabic (MSA + 4 dialects), English · .xlsx / JSON |
| Code repositories | 50M+ line corpus, plus production repos with full PR and commit history | 9 languages plus frameworks — C#, React, TypeScript, Angular, Java, Go, PHP, Swift, BrightScript |
| News & journalistic | 975,000-article archive (~672.75M words, 1999 onward) + live feeds | English, Spanish, Arabic · corpus and recurring feed |
| Clinical Q&A & cases | Golden and silver tiers, volumes scoped on request | English · .xlsx, PDF · de-identification scoped in |
| Raw & web-scale corpora | 20M words (~40M tokens) + 65,000 prompts + 11,000 TTS prompts | Arabic MSA and dialects · TXT / JSON / .xlsx |
| Documents & scanned text | 7,200 Serbian documents · 5,000 US tax filing packages | Serbian (Cyrillic + Latin), English · PDF |
| Academic & publisher | 1,328,228 journal articles · 4,137 book titles · 12 subject areas | English · API delivery |
| Scripted entertainmen | Scripts from a 90,000+ hour catalogue, 9 genres | English · scripts / transcripts (text) |
Conversational Q&A and instruction data
Eight human-authored sets covering Arabic and English, from 10,000 Q&A pairs on agriculture in Modern Standard Arabic through to 30,000-pair Saudi and pan-Arab collections. Dialect coverage spans Saudi, Emirati, Egyptian and Levantine, with domain sets for STEM and food. Everything ships as .xlsx or JSON with a consistent field structure, which matters more than it sounds: teams routinely lose a week reformatting instruction data before it reaches a fine-tuning pipeline. Full specifications sit on the conversational Q&A dataset page.
Know moreCode and software repositories
Production codebases across C#, React, TypeScript, Angular, Java, Go, PHP, Swift, JavaScript and BrightScript. Several carry their full change history — the industrial IoT frontend alone holds 141,471 lines across 306 pull requests and 1,603 commits. There is a general-purpose corpus above 50 million lines for pretraining, and a set of proprietary private-repository data at 8,632,700 tokens that has not appeared in public training material. If you are training a code-review model, the code dataset page is the one to read first.
Know moreNews and journalistic text
A 975,000-article English archive running from 1999, carrying byline, date, headline, body and topic tags under a controlled taxonomy from 2011 onward. Ongoing feeds add roughly 30,000 English, 16,000 Spanish and 12,775 Arabic articles a year. Archive plus feed is the combination that makes temporal reasoning and retrieval-augmented generation work.
Know moreClinical Q&A and case data
Medical question-answer sets, clinical case summaries and full-text articles, split into golden and silver tiers — golden held to a higher review standard for benchmarking, silver for bulk training. Redaction, de-identification and masking are scoped as part of the engagement rather than assumed.
Know moreRaw text and web-scale corpora
A 20-million-word Arabic corpus (~40M tokens) across MSA and regional dialects, a 65,000-prompt Arabic prompt set for teams generating their own completions, and 11,000 Saudi TTS script prompts.
Know moreDocuments and scanned text
7,200 real Serbian documents in Cyrillic and Latin script, and 5,000 US scanned tax filing packages. Both in authentic layouts and genuine scan quality — not synthetic renders, which is where most document AI models quietly overfit.
Know moreAcademic and publisher text
1,328,228 journal articles and 4,137 book titles across twelve physical sciences and engineering subject areas, from astronomy and biomedical through nanotechnology and polymer science, delivered via API.
Know moreScripted entertainment and narrative text
Scripts and transcripts drawn from a catalogue of 90,000+ hours of feature film, television, animation and vertical microdrama across nine genres. Text volume is confirmed at scoping, since the hours figure describes the underlying video library.
Know moreWorking out which of these fits your training run?
Where these datasets are not the right fit
The honest answer is that licensed data is not always the answer. If your model needs a narrow, proprietary behaviour that exists only inside your own product, custom collection will beat any off-the-shelf corpus. If you need exclusivity, note that these are non-exclusive licences — your competitor can license the same material. And volume alone rarely fixes a model; in our delivery work, the teams that improve fastest are the ones that diagnose the specific failure mode first, then buy narrowly against it.
How Shaip can help
Sourcing training text that is both legally clean and actually targeted at your model’s weak spot is slow work, and most teams underestimate it. Shaip’s Data Catalogs & Licensing practice removes that step. Our text data catalog offers licensed text datasets for AI training across all eight categories above, scoped to your project by language, domain, volume or format, and backed by Shaip’s data collection and annotation teams when off-the-shelf material needs extending. Samples or evaluation access can be arranged before you license, delivery runs through structured files, PDFs, recurring feeds or API, and PII and PHI handling is scoped into the engagement rather than left to you.
Building an Arabic assistant, a code-review model or a clinical reasoning system and unsure which corpus actually closes your gap? Share your requirements and our team will scope it with you.
