From 11 minutes to under one

In part 1 we ran one of Google's Agent Development Kit (ADK) examples inside our product. In part 2 its agents became a Cost Advisor that reads a Google Cloud bill and the official price list. This last part adds the team that turns findings into savings, and then the part that matters most in production: making the whole thing reliable and fast.

The first complete version took 11 minutes for one answer and sometimes lost results on the way. The final version answers in under a minute, the same way every time, and stops cleanly when Google's capacity is busy. Here is how we got from one to the other.

Step 3: a generator and a critic

The last agents of the example became the recommendations team:

• a savings generator turns the spend analysis and the pricing findings into concrete savings, each with an estimate per month, the effort, the risk and the condition under which it applies;
• a savings critic checks every proposal against the numbers the other agents found: is the arithmetic right, is it like-for-like (the same machine family, region, unit and project), is anything counted twice? It corrects what it can, drops what it cannot support, and writes the final report.

The critic's report has a fixed shape, so the product can use it and not just print text:

class Saving(BaseModel):
    title: str
    project: str
    service: str
    action: str
    monthly_saving: float = Field(ge=0)
    basis: str           # how the estimate is computed
    effort: Literal["low", "medium", "high"]
    risk: Literal["low", "medium", "high"]
    conditions: str

class CostOptimizationReport(BaseModel):
    currency: str
    monthly_spend: float
    savings: list[Saving]
    total_monthly_saving: float
    dropped: list[str]   # what the critic removed or corrected, and why
    summary: str

Our Agent Lab page reads that report and shows it as a table: the monthly spend, the possible saving, each saving with its effort and risk and how it is computed, and what the critic removed.

The critic earned its place in the first runs. It dropped a commitment proposal for a database because no commitment price had been found, saying so instead of guessing. It turned ranges into one checked number. And it refused to count two savings on the same machines twice (switching them to spot and scheduling them off at night).

The first full run took 11 minutes

The first version with all the agents worked, and took 661 seconds. Measuring each step showed where the time went:

• Every hand-off cost 23 to 38 seconds. In that version a coordinator agent decided who worked next, and every transfer_to_agent is a full model call. Gemini 3 models think before they answer, and nothing limited how long.
• Specialists writing their findings took 77 to 118 seconds, for the same reason, plus long answers.
• The first price lookups took 45 to 84 seconds: the price lists of the big services are large (Compute Engine alone is about 33,500 SKUs).

Two changes brought it to 91 seconds: a thinking level per agent (LOW for the agents that route or collect, a bit more for the one that checks), and reading the big price lists when the service starts, in the background, so the first question does not wait for them.

config = types.GenerateContentConfig(
    thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel("LOW")),
)

One detail: the model we use rejected the lowest level (MINIMAL) with "Thinking level is unsupported", so LOW was the floor.

Results that disappeared

In ADK an agent saves its answer to the session state under its output_key, and the next agent reads it from there. Some runs ended with an empty state for one of the agents, and the agents after it worked half blind.

The cause is in ADK's source: an answer is saved to output_key only when it is final, meaning it contains no function call. A specialist that wrote its findings and called transfer_to_agent in the same answer never had them saved.

The first fix, an after_agent_callback that saves the text, did not work either: with transfers, the next agent runs inside the current agent's turn, so the callback only ran at the very end of the whole run. An after_model_callback, which runs the moment the model's answer arrives, worked. Then, with less thinking, the spend agent once handed over without writing anything at all.

That was the real lesson. Letting the model decide the order of steps that never change is fragile and slow.

Version 2: the order lives in code

The second version is simpler. There is no coordinator. The root is a SequentialAgent that always runs the same four agents in the same order:

root_agent = SequentialAgent(
    name="cost_advisor",
    sub_agents=[spend_analysis_agent, pricing_agent, savings_generator_agent, savings_critic_agent],
)

No model call decides the order and nobody transfers, so every answer is final and ADK saves every result. The workaround callback could go. The graph in Agent Lab became a straight line: four agents, three hand-offs, always the same.

Use model-driven routing when the questions vary and the model must choose the path. Use workflow agents (SequentialAgent, ParallelAgent, LoopAgent) when the path is known.

Time limits that stop a run

While measuring version 2, Gemini's newest Flash model was overloaded. Even a request of two words ("Say OK.") came back with 429 RESOURCE_EXHAUSTED or 504 DEADLINE_EXCEEDED after almost a minute. An agent that waits for that hangs, and a user who waits for an agent leaves.

So every run now has three limits:

The call limit is a setting of the request (HttpOptions(timeout=...)). The agent and run limits are a small watchdog next to the run: every second it checks how long the current agent and the run have taken, and past a limit it cancels the run's task. Cancelling also cancels the Gemini call in flight. The page shows Stopped and the reason, for example "pricing_agent took more than 90 s, so it was stopped."

The structured output that took three minutes

With the limits in place, the runs that failed all failed at the same step: the critic. Its single call ran past the limit every time, even with the lowest thinking level.

Timing that one call on its own, with the same inputs, found the cause:

Constrained decoding, which forces the model's output token by token into the schema, was what made the call so slow for this report. Asking for JSON in plain JSON mode, listing the fields in the instructions and validating the answer with the same Pydantic model in an after_model_callback gave the same valid report about twenty times faster. When an answer does not validate, nothing is saved under a wrong shape.

A fallback model

Retries help when a model is briefly busy. They do not help when it stays busy, as it did that evening. Newly released models are often the busiest, and the previous generation usually has capacity.

The main model is now the newest Flash; when it is still busy after one retry, or a call passes its time limit, the same request (as the agent built it, before the first try changed it) goes to the previous Flash model, which has its own retry. It is a small subclass of ADK's Gemini model class:

class FallbackGemini(Gemini):
    fallback: Gemini | None = None

    async def generate_content_async(self, llm_request, stream=False):
        backup = llm_request.model_copy(deep=True)
        try:
            async for response in super().generate_content_async(llm_request, stream):
                yield response
            return
        except errors.APIError as e:
            if e.code not in (429, 503, 504):
                raise
        backup.model = self.fallback.model
        async for response in self.fallback.generate_content_async(backup, stream):
            yield response

Agent Lab shows which model answered each step, so a fallback is visible when it happens. Errors that are not about capacity, such as a bad request, are raised as before.

Paying for reserved capacity (Provisioned Throughput) is the other answer, for products with steady traffic. Hosting your own model is rarely the answer for this kind of work: open models are weaker at multi-step reasoning like the critic's, and GPUs cost money whenever they are on.

Before and after

The same question on the same sample billing data:

What to consider

• Measure each step before optimizing. Our slowest part was not where we expected: first the hand-offs, then one structured-output call.
• Put fixed order in code. Workflow agents are faster, cheaper and always complete; keep model routing for open questions.
• Know when ADK saves state. output_key is saved only from a final answer, one without function calls.
• Limit thinking per agent. Routing and collecting need little; checking needs a bit more.
• Time-box every call, every agent and every run, and tell the user when a run stops and why.
• Test structured output on its own. Constrained decoding can be much slower than JSON mode plus validation in code.
• Plan for busy models. Retries for short bursts, a fallback model for longer ones, reserved capacity when you have real traffic.
• Add a critic. A second agent whose only job is to check the first one's numbers catches the slips no prompt rule fully prevents.

Where this goes next

The Cost Advisor now does one job well. The same building blocks scale to more than one job: next, the company behind cloudcost.info gets its own team of independent agents, each deployed on its own on Agent Runtime, with its own identity, its own tests and one human reviewing their work.