For the past few months, I’ve been working on an AI agent for antibody optimisation. The idea is to let the agent use existing computational tools to identify developability issues, suggest modifications, and evaluate the resulting candidates.

Evaluating such an agent might sound straightforward: is the optimised antibody better than the starting point? But obviously, we cannot run wet-lab experiments for every generated antibody. And what if the agent modifies a sequence and returns a reasonable-looking candidate, but the modification was based on what the agent thought might work, without any supporting analysis? Would we accept that?

Traditional LLM benchmarks often have a known answer. Agents are different: they select tools, interpret results, make decisions, and execute multi-step workflows.

Scientific tasks are particularly challenging because they are often open-ended, with multiple valid solutions and competing objectives. There may be no single correct way to optimise an antibody, and computational success does not necessarily imply biological validity.

So how do we evaluate an agent when we don’t know the best answer and cannot experimentally validate every result?

One approach is to evaluate the process, not just the outcome. This is explored in benchmarks such as BiomniBench, which uses expert-defined rubrics to evaluate biomedical research agents, and TRAJECT-Bench, which evaluates tool selection, arguments, and execution order separately.

For my project, I found it useful to record the agent’s tool calls, inputs, outputs, and intermediate results, and divide the evaluation into four parts:

1. Tool execution: Did the tools run successfully, with valid inputs? Did the agent recognise and handle errors? It’s also important to test edge cases and check whether the agent understands the tool’s parameters and knows how to set them appropriately for different tasks.

2. Tool selection: Did the agent choose the appropriate tools? For example, if I ask it to humanise an antibody, does it know to use a tool such as Hu-mAb?

3. Scientific interpretation: Given the correct tool output, did the agent draw the right conclusion? This is probably the most challenging part to evaluate, as it requires domain expertise to assess whether the agent interpreted the results correctly and made scientifically justified decisions.

4. Multi-step workflows: Since I expect the agent to handle longer workflows, I wanted to test whether it could complete different multi-step tasks. Can it translate a request into a sequence of actions, execute the steps in the correct order, and use the results to decide what to do next?

The first three levels help us evaluate individual capabilities, but ultimately we want the agent to combine them to complete more complex tasks.

What if the instruction is simply “Optimise this antibody”? There may be dozens of valid mutations and several reasonable strategies, and we might not know what the best solution should look like.

In our case, we decided to define a specific workflow and evaluate whether the agent follows it correctly. Breaking complex workflows into smaller tasks that can be evaluated separately also makes debugging easier.

Of course, this is not necessarily the right approach for every scientific agent. Sometimes we might want the agent to explore different strategies and decide which tools to use. How much freedom we give it probably depends on the scientific question we’re trying to answer.

Either way, we can define constraints that the final result must satisfy. We might not know which solution is best, but we can still check whether the agent respected the constraints we defined, for example, preserving the CDRs during antibody optimisation.

A few things I found useful when developing evaluations:

• Decide where to check the agent’s progress. Identify points in the workflow where you want to verify that the required steps were completed. The Anthropic SDK’s tool runner, for example, lets you inspect proposed tool calls before execution.
• Use deterministic checks wherever possible. Anything that can be checked directly in code should not require another LLM to judge it. For example, verifying output formats, checking numerical thresholds, or ensuring that predefined constraints are satisfied. These checks are simpler, cheaper, and more reproducible.
• Use structured rubrics for open-ended tasks. Define what a successful result needs to satisfy without requiring one specific solution.
• Version your evaluation setup. Prompts, tools, and workflow instructions all affect performance. Interestingly, BiomniBench reports that changing the agent harness can have a larger effect on evaluation scores than moving between successive model generations. Version them and report them with your numbers.

I think one of the biggest challenges in evaluating scientific agents is that they can perform many different tasks and easily produce convincing-looking results, even when the decisions behind them are wrong. Breaking evaluations into smaller components helps us understand whether the agent is using tools correctly, following the intended workflow, and making decisions supported by the results.

Of course, none of this guarantees better results (in our case, better antibodies). That still depends on the quality of the tools, the agent’s decisions, and ultimately experimental validation. But good evaluations help us understand what is failing and what we need to improve next.