Evals for AI and search: know if a change helped.
Without evaluation, every model change is a guess. We build offline metrics, LLM-as-judge pipelines and online experiments that your team trusts and keeps running.
Sound familiar?
- Releases are judged on gut feel and a handful of spot checks.
- Offline metrics say one thing and online results say another.
- Nobody trusts the LLM-as-judge scores.
- A/B tests come back inconclusive or take too long to read.
- Fixes for one problem quietly break something else.
How we fix it.
Metric design
Offline metrics that track the outcomes your business actually cares about.
Golden sets & guidelines
Representative test sets and labelling guidelines that people apply consistently.
LLM-as-judge
Judge prompts and rubrics, calibrated and checked against human labels before you rely on them.
Regression suites
Automated checks in CI so quality can't silently drop between releases.
Online experiments
A/B test design with the right sample sizes, guardrail metrics and readouts.
Deliverables your team keeps.
Common questions.
Can LLM-as-judge replace human labels?
Partly. A judge calibrated against a sample of human labels can scale evaluation cheaply. We measure how often it agrees with people before anyone relies on it.
How much labelled data do we need?
Often a few hundred well-chosen examples is enough to start. We grow the set where the metrics turn out to be noisy.
Can you evaluate both search and LLM features?
Yes. Ranking, retrieval and generated answers each need different metrics, and we design them to work together.
Will our team be able to maintain it?
That's the goal. Everything lives in your repository and is documented, and we hand it over with your engineers.
Often combined with.
Search relevance & ranking
Fix the queries that return the wrong results: query understanding, hybrid retrieval and learning to rank, measured end to end.
Learn moreRAG & retrieval
Make LLM answers right by fixing what they retrieve: chunking, embeddings, hybrid search, reranking and RAG evals.
Learn moreLLM pre-training & fine-tuning
Make an open model fluent in your domain with DAPT, SFT and DPO, from data curation to a model you can ship.
Learn moreHave a search or LLM problem worth solving?
Tell us what you're building. You'll talk directly with the founder, not a sales team. If we're not the right fit, we'll say so and point you to someone who is.