How to Hire AI Developers
Establish which of three distinct roles you need: someone building product features on existing models, someone running models reliably in production, or someone doing research. They share a title and share almost no skills. Then screen for evaluation and failure handling rather than model knowledge.
Decide which role you need
Application engineer. Builds product features using existing models through APIs. The work is prompt and context design, retrieval, tool integration, evaluation, cost control and handling the cases where the model is confidently wrong. This is what most companies actually need and it is the role most frequently mis-specified.
Systems or platform engineer. Runs models reliably: serving, latency, throughput, caching, routing, and sometimes fine-tuning or hosting. Closer to infrastructure engineering than to machine learning, and required once usage is large enough that inference cost or latency is a real constraint.
Researcher. Trains or substantially modifies models. Necessary if your product is the model. Unnecessary and expensive if your product uses models, which is the far more common case.
The common failure is writing a posting listing research credentials for a job that is application engineering. You then interview researchers for a role they will find unengaging, hire one, and lose them within a year to work that uses what they trained for.
Write down what this person does in their first month. If the answer is shipping a feature that calls a model, you need the first role, and the posting should say so.
What to screen for
Evaluation discipline. The single most predictive signal. Ask how they knew a system was working. Strong candidates describe a test set, a rubric, regression checks and a way of catching quality drift. Weak candidates describe trying it and it seemed good. This distinguishes people who have shipped from people who have demoed, and it is a distinction that matters enormously in production.
Failure handling. What does the system do when the model returns nonsense, times out, or produces something plausible and wrong. Someone who has run this in production has opinions and scars. Someone who has not will say the model is usually right, which is not an answer.
Cost awareness. Whether they can explain what drives spend and what they did about it. Not because you are cheap, but because it demonstrates they operated something at a volume where it mattered.
Context and retrieval judgement. Knowing when to retrieve rather than to stuff a prompt, and why irrelevant context degrades quality as well as raising cost.
Domain scepticism. Willingness to say a use case is a poor fit for a model. Candidates who think everything is solvable this way will build things that fail expensively and confidently.
What to weight less: familiarity with a specific vendor or framework. Those change quickly, and someone with the judgement above learns a new one in a week.
The work sample that works
Give them something broken. A prompt or pipeline that produces wrong output on some inputs, and ask them to diagnose it. This surfaces exactly the skills the role needs, cannot be prepared for, and takes an hour. It is far more informative than asking someone to build something from scratch, which mostly tests whether they have built that thing before.
Where to find them and what it costs you
The supply picture is worth being realistic about. People with genuine production experience are scarce because the work is recent, so a strict experience requirement filters your pool to almost nobody and mostly filters for people who were early rather than people who are good.
The more available and often better candidate is a strong engineer who has been building with these tools seriously for a while. Judgement about systems transfers; the model-specific parts are learnable and the engineering discipline is not.
Which means the search widens considerably if you screen for evaluation habits and failure thinking rather than for years in a title that did not exist recently.
On evidence: this is a field where public work is unusually informative, because people building with these tools tend to write about what broke. A candidate with a public record of shipping and debugging real systems is easier to assess than one with a resume claiming the same, and a hiring process that can check the claim spends its interview time on judgement instead of verification.
Be specific in the posting about which of the three roles it is, what the system does, and what the hard part is. Vague AI postings attract volume and the wrong volume.
Frequently asked questions
- What should I look for when hiring an AI developer?
- First establish which of three roles you need: application engineering on existing models, systems work running them reliably, or research. Then screen for evaluation discipline, failure handling, cost awareness and retrieval judgement rather than familiarity with a particular vendor or framework.
- What is the most common mistake in hiring for AI roles?
- Writing a posting that lists research credentials for a job that is application engineering. You interview researchers for work they find unengaging, hire one, and lose them within a year. Write down what the person does in their first month and specify accordingly.
- What is the best interview signal?
- How the candidate knew a system was working. Strong answers describe a test set, a rubric, regression checks and a way of catching quality drift. Weak answers describe trying it and it seeming good, which separates people who shipped from people who demoed.
- Should I require production AI experience?
- A strict requirement filters your pool to almost nobody, and mostly selects for people who were early rather than people who are good. A strong engineer who has been building seriously with these tools usually transfers well, since the engineering discipline is the hard part and the model-specific knowledge is learnable.