Three bugs in my agent pipeline that never logged an error

Kyle had been running for months. No alerts, no error rate, no complaints. Then I wrote a set of deterministic checks over its output and found three separate failures, each of which had been happening on every single request.

None of them threw.

The scraper returned nothing and called it success

The researcher agent searches DuckDuckGo's HTML endpoint and scrapes the top three results. The parsing worked. The selectors matched. Ten results came back in the document.

Then this line threw all ten away:

if exists && !strings.Contains(rawURL, "duckduckgo.com") {
	results = append(results, SearchResult{Title: title, URL: rawURL})
}

The filter was written when DuckDuckGo linked directly to results and wrapped only its own ads. At some point it started routing every outbound link through its redirector:

//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.postgresql.org%2F...

Every href now contains duckduckgo.com, so every result was discarded as internal. The function returned "No results found." with a nil error. The writer received the string "None" as its source context and fell back to model priors, which it is designed to do when research genuinely finds nothing.

So the pipeline reported a researcher stage that ran successfully, took eight seconds, and contributed nothing. Documents that claimed to be grounded in live sources were written entirely from training data. The headline capability of the project had stopped working and the logs were clean.

Unwrapping the redirector is six lines. The second change matters more: an empty result set is now recorded as a failure.

case len(found) == 0:
	log.Printf("Scraper returned no usable results for %q", args.Query)
	research.Finish(telemetry.OutcomeRetryableFailure, telemetry.ErrScraperFailure)

A search that parses correctly and yields nothing is not a success. It was that "success" that hid the bug for months. Now it lands on the failure taxonomy where somebody can see it.

Gemini 3.x wants a signature it did not tell me about

Around the same time, every topic that triggered a web search started failing at the planner with a 400:

Function call is missing a thought_signature in functionCall parts.

Gemini 3.x returns an opaque thought_signature alongside a tool call, and requires you to send it back when you replay that call in the next turn. My ToolCall struct did not model the field, so it was dropped on deserialisation and the follow-up request was rejected.

This one at least had the courtesy to fail loudly. What made it nasty is that it only affected the search path, which was the path the other bug had already made pointless. Two independent faults on the same code path, each masking the significance of the other.

The fix is to stop modelling the field:

// ExtraContent is provider-specific data attached to a tool call that must be
// echoed back verbatim when the assistant turn is replayed.
ExtraContent json.RawMessage `json:"extra_content,omitempty"`

Raw JSON, round-tripped without interpretation. A narrower type would be a narrower thing to break the next time a provider adds a field. I A/B tested it against the live API before believing it: 400 without, 200 with.

The planner made a different decision on the same input twice

The planner decides whether a topic needs research. It has two tools, web_search and skip_search, and I force it to call one of them. On a topic about comparing multi-agent and single-agent pipelines, it called skip_search and wrote a confident document with no sources. An earlier run on the identical topic had called web_search.

Same prompt, same model, different answer.

The cause is that GenerateComplex ran at temperature 0.3, which is a reasonable setting for prose and the wrong setting for a classification. I was sampling a decision that should have been deterministic.

Three changes, ordered by how much they matter:

Temperature 0 for the routing turn only. The writer keeps 0.3, because that one is generation and benefits from it.

An allowlist instead of a description. The old skip_search description explained when skipping was appropriate, and the model talked its way into it with "conceptual architecture and trade-off analysis, requiring no external temporal facts". It now enumerates the only three permitted cases: a pure definition, pure arithmetic, or an explanation of code supplied in the prompt.

A deterministic backstop underneath both. If the topic contains a comparison marker, a superlative, or a temporal marker, skip_search is not offered as a tool at all. A tool that is not in the list cannot be chosen, whatever the sampling does.

That last one matters more than it looks, because one of my two providers cannot run at temperature 0. Kimi rejects anything but 1 outright. On that provider the routing decision is still sampled, and the backstop is the only guarantee there is.

Then I ran the decision ten times on each of seven topics, three real and four sitting near the boundary. Every topic that returned an answer returned the same answer all ten times. One green run would have proved nothing about a stochastic decision.

What I take from this

All three failures shared a shape: the code did something reasonable, returned a value that looked valid, and moved on. Error rates cannot see that. Uptime cannot see it. The only thing that caught them was asking a different question, which is not "did it fail" but "does the output have the properties it is supposed to have".

The checks themselves are unremarkable. Did a run that fetched sources produce at least one citation. Does every cited source id resolve to something the researcher actually retrieved. Did a run perform research, or state a reason for not doing so. Fourteen of them, all pure functions over the document and its telemetry, no model calls, no network.

They run in under two seconds and found three months of breakage on the first execution.

If you are running an agent pipeline and your only signal is that requests return 200, you are in the position I was in. The output is probably fine. You do not currently have a way of knowing.