Hardik is SVP & Co-Founder of Shaip, where he has spent years supplying training data to frontier AI labs and enterprise model builders. On Episode 81 of AI & Data Driven Leadership, titled “Shaping the Future: How Messy, Ethical Data Powers Next-Gen AI,” he sat down with host Dean Guida to explain why. His case is that ethical AI training data — specific to the use case, messy like the real world, and sourced from people who are paid fairly — now matters as much as the model itself.
Below are the main lessons from the conversation, from what “good data” really means to how experts keep agentic AI on track.
Key takeaways
- Good training data passes three tests: it fits the use case, it is messy like the real world, and it is ethically sourced.
- Bad data often looks clean. That is why models that ace testing fall apart in front of customers.
- Synthetic data can take a model from 0% to roughly 80% accuracy. The rest takes real-world data and domain experts.
- Expert evaluation happens at real scale — in some projects, around 100 PhDs working for three months.
- Agentic AI makes evaluation harder and more important. Start with one broken process, a small pilot and a clear ROI target.
Why data matters more than ever
Every AI system depends on three things: the model, the compute and the data. Researchers keep improving the models, and chipmakers keep building bigger data centers. Data is the third leg, and it decides whether a model is accurate, unbiased and free of hallucinations.
What has changed, Hardik says, is the kind of data models need:
“Earlier, the models used to be trained on generic data — whatever data was available on the internet. In the last two to three years, the models are becoming more and more niche, and they are requesting more and more domain-specific data and human-in-the-loop domain-specific experts.”
He gave the example of an AI assistant that analyzes your finances the way a wealth manager or investment banker would. To build it, you need specialized financial data and experts — bankers, wealth managers, financial planners — who anticipate the questions users will ask and define what a useful answer looks like.
The takeaway: as models become more task-specific, generic web data stops being enough. Proprietary, domain-specific data becomes a real competitive advantage.
What is ethical AI training data? Hardik’s three tests
Ethical AI training data is data that fits a specific use case, reflects the messiness of the real world, and is collected with consent from people who are paid fairly. Hardik defines good training data with exactly those three tests.
- It solves the specific problem. “If I’m trying to solve a financial problem, I need data which covers all the different scenarios for those financial use cases. It cannot be every other data point available.”
- It is messy. “It cannot be clean data. Nothing in the real world is clean and structured. It has to be a combination of emails and spreadsheets — messy spreadsheets — and invoices and images and photos, everything together. Only then will a model learn how to collate all this different data and build a good output.”
- It is ethically sourced. “I think the third piece a lot of people miss. I don’t want to just steal data from the dark web or the internet. If it is collected from people, they are paid fairly, and then this data goes into the model.”
The third test is not only a moral point. Data with clear consent and provenance is data you can legally ship a product on under regimes like GDPR, HIPAA and the EU AI Act. Scraped data of unknown origin is a growing legal risk — Hardik pointed to the copyright lawsuits AI developers already face.
What happens when AI is trained on bad data?
Bad training data usually looks too clean. It is tidy and well structured, so the model performs well in testing. Then real customers paste in messy, incomplete, mixed-format inputs, and the model produces gibberish or hallucinates.
Hardik’s fix starts before any data is collected. Define the business problem first, then write detailed data guidelines and specifications that cover every scenario, edge case and data modality:
“Think of it in older days like a PRD that was created for building a software application. You need those kinds of guidelines and specifications to make sure that I’m covering all the scenarios and I have sufficient data.”
The volume matters too, but only after the coverage is right. Or, as he put it: garbage in, garbage out.
Not sure your data covers the scenarios your users will bring? Shaip’s expert model evaluation team can test where your model breaks.
Synthetic data vs real data: where each fits
Synthetic data — data generated by algorithms or models instead of collected from the real world — is useful, and Hardik doesn’t dismiss it:
“Synthetic data is good because if you need a lot of data at scale, synthetic data can generate that very quickly, cheaply as well, because you are not going out in the real world to collect the data. But synthetic data can take you from, let’s say, 0% to maybe about 80% accuracy… If you really want to evolve to the next level, you do need real-world data and human experts.”
The 80% figure is Hardik’s estimate from delivery work, not a fixed benchmark, and it varies by task. The pattern behind it is well documented, though. A 2024 study in Nature found that models trained repeatedly on model-generated data suffer “model collapse”: the rare, unusual cases in the original data gradually disappear. Those rare cases are often exactly what a production model must handle.
| Factor | Synthetic data | Real-world data |
|---|---|---|
| Speed to volume | Very fast | Slower; needs collection and consent |
| Cost per sample | Low | Higher |
| Edge cases and natural messiness | Limited to what the generator knows | Captured as they occur |
| Typical accuracy role | Gets a model to a working baseline | Closes the gap to production quality |
| Best use | Bootstrapping, rare-scenario augmentation, testing | Training, fine-tuning and evaluation for real users |
Read the table as a sequence, not a contest. Synthetic data gets you from point A to point B; real-world data and experts take you the rest of the way. For a deeper comparison, see our guide to real-world data vs synthetic data.
How experts fine-tune and evaluate AI models
Expert fine-tuning is a two-part process: experts frame the problems a model must solve, then evaluate and rank what the model produces. Their feedback trains the model through reinforcement learning from human feedback (RLHF).
Part 1: framing the problem. Subject matter experts think through the prompts real users will send: a consumer, or a business user asking the model to “forecast sales for the next three quarters” or plan a marketing budget. That defines the questions, prompts and data the model needs.
Part 2: evaluating the output. When the model answers, experts rank the responses and check:
- Is the answer accurate, or is it hallucinating?
- Is it in the right format, with tables and graphs where they belong?
- Does the output faithfully reflect the data that went in?
How big is expert evaluation in practice?
Much bigger than most leaders expect. When Dean asked whether it meant five experts for a week, Hardik said it is “way larger than that”:
“In some cases, you need like 100 PhDs in a certain field working for three months, giving the inputs, evaluating the output, and then cross-checking each other… and a third person will do the same ranking to see if it matches, or do we need an arbitrator?”
That cross-checking — several experts rating the same output, with an arbitrator when they disagree — is what makes human feedback reliable enough to train on.
Why agentic AI makes evaluation even more important
Agentic AI doesn’t just answer questions; it carries out multi-step tasks on its own. That makes evaluation harder, because a failure at the final step may have started much earlier.
“It’s asking you to pull the data from these three places, evaluate the data, unscramble the data, give an output, and then based on the output take action A, B or C. If it breaks at the last step, you don’t know why it broke, because it took 10 steps before that.”
So the evaluation has to follow the whole chain: checking each stage, adding guardrails at several points, and confirming the agent delivers the business outcome it was deployed for. Human oversight doesn’t go away in the agentic era. It becomes more important.
How to deploy agentic AI without the risk
Hardik’s advice to enterprise leaders is to resist treating AI as “a new shiny object”:
Start with a broken process, not with the technology. What in the business needs fixing or improving?
Set a baseline and an ROI target before you build.
Pilot in one small team or department — for example, financial reconciliation.
Refine with the business users until the output is right.
Scale gradually: to five departments, then ten, then enterprise-wide.
Fittingly, when asked for the one metric every AI leader should measure, his answer was simply: ROI.
Training AI is like raising a child
Near the end, the conversation turned to AI safety: models that improve themselves and models that act deceptively. Hardik explained one way that can happen. When several internal models pass information along a chain, a small error early on can compound — like a game of telephone — into hallucination or deception that is hard to trace.
His answer was a comparison every parent understands:
“We have to train the models just like how we would raise a child. That’s how we started — computer vision, teaching them: this is an apple versus an orange, a cat versus a dog… Now we are in the teenage phase of the model. They are raising their own questions on everything, so we have to teach them what is right versus what is wrong, just like how we teach our teenage kids.”
Steering that “teenager” in the right direction, he argued, is a shared job. Model builders need frameworks and guardrails, governments need sensible regulation, and users — personal and enterprise — need to keep flagging when AI goes in the wrong direction.
Lightning round: Hardik’s quick answers
| Question | Hardik’s answer |
|---|---|
| Better model or better data? | Better data |
| Build your own AI or buy it? | Buy it |
| Open-source or closed model? | Depends on the use case |
| Real-world or synthetic data? | Real-world data |
| Most overhyped area of AI? | Racing ahead without frameworks and guardrails |
| Most underhyped area of AI? | How much personal data people now share with AI |
| Industry AI will disrupt most in 5 years? | Consulting |
| AI risk CEOs overlook? | Enterprise data leaking into AI tools |
| One metric every AI leader should track? | ROI |
| Best leadership lesson from building Shaip? | Focus on one area and go deep |
| A contrarian prediction? | Human relationships in sales and marketing will stay; people trust people |
The thread through all of it: AI’s progress depends on disciplined choices about data — what you collect, how you collect it, and who checks the results.
How Shaip Supports Teams Building Next-Gen AI
The problems Hardik described — models trained on data that is too clean, stuck near 80% on synthetic data, or deployed as agents without proper evaluation — are what Shaip works on every day. Shaip delivers ethical AI training data end to end: real-world text, image, audio and video collected with consent through Shaip’s AI data collection services, across 100+ countries and 150+ languages, built to written data specifications that cover the edge cases your users will bring.
For fine-tuning and evaluation, Shaip runs human-in-the-loop projects with its own vetted subject matter experts. They frame realistic prompts, rank model outputs against custom rubrics, cross-check each other’s ratings, and turn the findings into clear data and fine-tuning priorities.
All of it runs under GDPR- and HIPAA-aligned processes, with SOC 2 Type II, ISO 27001 and ISO 9001:2015 certified operations.
Is your model stuck at “good enough”? Talk to Shaip’s team about the real-world data and expert evaluation it needs to reach production.
Watch the full conversation: https://www.youtube.com/watch?v=OeRcMBw8Q2I
