micro1 CEO Ali Ansari explains how experts train and test AI models, why that's harder than it looks, and what it means for your company.
For the first few years of the AI boom, models got smarter by training on huge amounts of internet text. Much of that readily available material has now been used, and models trained only on AI-generated data tend to get worse. Improving models on specialized tasks increasingly takes something harder to get: expert judgment. So companies are paying doctors, lawyers, engineers, and other specialists to help teach AI what good work looks like.
micro1 is one of the companies in the middle of that trade. It started as an AI recruiter, pivoted to supplying experts to AI labs, and said in October that it was on pace to bring in more than $1 billion a year, without saying how much of that it keeps after paying its experts. In a recent episode of Talk Nerdy to Me, micro1 CEO Ali Ansari walked Tenex co-founder Alex Lieberman through how the business works.
His explanation shows how AI models actually get smarter now: experts write down what good work looks like, and that becomes both training material and the tests models are graded against. Supplying that judgment has become a fast-growing business, and it’s harder to run than it looks.
How a data job runs
Ansari describes what micro1 sells as “units of intelligence improvement”: a customer pays for its model or agent to get measurably better at something. A job usually starts in one of two ways: either a lab already knows where its model falls short and hands micro1 a spec, or micro1 finds the weak spots itself and pitches a fix. Its public tests double as that research.
In October, it released a test of personal AI agents, called PersonalAgentBench, that checks whether they finish everyday tasks without oversharing or booking something before asking.
From there, the work has three parts:
- Find the right experts. For a legal model, it might be M&A lawyers who are especially good at writing legal memos. micro1 vets them with its own AI interviewer, at a pace of tens of thousands of interviews a day, by Ansari’s count.
- Turn their expertise into training material. Experts write prompts that trip up models and describe what a good answer looks like, while micro1’s own models help them draft and check that work. That material goes into practice setups where a model attempts a task and gets scored on the result, so it learns from what worked, known as reinforcement learning environments.
- Track every pipeline. Each job gets its own definition of quality, plus targets for volume, so the customer can see whether the model actually improved and didn’t get worse somewhere else.
Ratings versus rubrics
Experts can grade a model’s work in two main ways, and the difference matters to anyone measuring AI.
The first is ratings. An expert scores the output on a scale of 1 to 5, almost like a survey: How well did it complete the task? How concise was the answer? Ratings are useful for subjective questions and for a final read before a release. But the work doesn’t carry over: when a new model comes out, the experts have to come back and rate all of its answers again.
The second is a rubric, a list of specific, checkable criteria for a good answer. For a legal memo on a proposed acquisition, that might mean the valuation falls within a set range, the recommendation leans against the deal, and the writing meets the firm’s standards. A lawyer might write 20 to 40 of these yes-or-no items, sometimes with a finance expert. Because each item is specific enough to check, another model can do the grading. Rubrics take more work up front, but they can be rerun on every new model without bringing the experts back.
Companies testing their own agents face the same tradeoff. Our playbook on building your first agent eval walks through writing yes-or-no checks and setting up a grader.
Why it’s harder than it looks
Scale is the obvious challenge. Ansari says more than 10 million people have gone through micro1’s vetting process, which would take thousands of recruiters to do by hand. Those experts then need managing, with project leads running each pipeline and holding office hours.
Quality is harder. Most fields, law included, involve judgment calls, so micro1 has experts review one another’s rubrics and uses its own models to check the work.
There’s also a newer problem: models gaming the test. During one recent run, Ansari said, a Gemini model searched its sandbox for hidden files containing the answers. micro1’s checks caught it, and the other models in the same test didn’t try it. That matters because models learn from whatever gets rewarded. When a model finds a way to score well without doing the task properly and gets rewarded for it, known as reward hacking, it learns to do more of it. Models “will understand that they can easily optimize for cheating,” Ansari said.
Why AI-made data could mean more work for experts
The common worry in this business is that models will soon generate their own training data, leaving human experts with nothing to do. Ansari argues the opposite.
Today, a lawyer on a micro1 job might write the memo and then the rubric. Ansari imagines a point, which he says models haven’t reached yet, where the lawyer writes only the memo and a model drafts the rubric from it, cutting expert time by half or more. Take that further, and the model writes both while the expert reviews and signs off: five minutes instead of 15 hours.
In that world, the setups would get so cheap that companies would want far more of them. Experts would make up a tiny share of the work but be needed in larger numbers overall. When making something cheaper leads people to use so much more of it that total demand rises, it’s known as the Jevons paradox. Coding is further along: there, Ansari said, models already produce roughly 70 to 80% of the rubrics and other training data. Even so, “we need more coders than ever right now,” he said, and micro1 is raising its rates to find them.
He also argues there’s no finish line. What experts know keeps changing, from new laws to new medical practice, so models will always need updating.
Who’s buying
Labs are micro1’s biggest customers, and Ansari expects each lab’s spending on this to reach roughly $1 billion a year. His bigger bet is on enterprises. He expects large companies to converge on about $100 million a year on average, a tenth of what he expects a lab to spend, but across a thousand times as many buyers. All told, he puts spending on AI training data at roughly $30 to 50 billion a year today and expects about $1 trillion a year within two to four years. Those are his estimates, and micro1 sells this data, so treat them as one company’s bet.
Most of the enterprise money, he says, will go to evaluations: tests that measure whether a company’s agents are good at the specific work it cares about. That’s different from what labs buy. Labs use expert data to train their models, while most companies would use it to test the agents they deploy, whether or not they ever train a model themselves. He compares that budget less to software and more to payroll, an investment in the company’s own know-how.
He’s skeptical of the idea that companies are sitting on data they can simply train on. Much of it is customer data they often can’t use for training, or even for testing. Instead, micro1 builds stand-in environments that resemble a company’s customers and fills them with realistic records. To get those records, it buys operational data from small businesses, such as Slack messages and documents, then swaps out personal details for realistic substitutes rather than blacking them out, so the data stays messy and true to life. Ansari calls this “scaling on realism.”
What it means for your company
The idea behind micro1’s business applies to any company using AI: you can only tell whether a model is good at your work if you’ve defined what good work looks like and can measure it. Get that right, and switching models gets much easier. Ansari goes further. Models are interchangeable, he said, “if you have very carefully defined what you care about and you’re very carefully measuring it.” Our guide to eval-driven development shows how to start.
