From Google's ADK recipe to a multi-agent in our product

I wanted to learn Google's Gemini Enterprise Agent Platform by building something real, not by reading about it. So I set myself a goal: take one of Google's ready-made agent examples, run it, and put it inside our own product, cloudcost.info, so a logged-in admin can start the agent from a web page and watch it work.

It took one day. Almost nothing went smoothly, and that is why this article exists. Below is what we built, every problem we met (with the real error messages), how we solved each one, and what I would think about before building an agent of your own.

This is part 1 of a series. In the next parts we turn this example into a Cost Advisor agent for cloud bills.

What we built

We started from Google's Agent Development Kit (ADK) recipe brand-search-optimization (in the open-source google/adk-recipes repository, Apache-2.0). It is a small team of agents that improves product titles for an online shop:

• a coordinator that receives the request and hands work to specialists;
• a keyword agent that reads the product catalog from BigQuery with a Python tool;
• a search agent that drives a browser with Gemini Computer Use to see what competitors write;
• a comparison agent with two sub-agents, a generator that proposes new titles and a critic that reviews them.

The shop example does not matter to us. What matters is the shape: a coordinator, specialists with tools, data from BigQuery, and a generator/critic pair. That is the shape of most business agents.

Here is how it runs in our product:

Browser: cloudcost.info/agent-lab (admin only)
   │  start a run, then ask for new steps every 2 seconds
   ▼
Flask API on App Engine   ← checks the admin login
   │  Google ID token (private service-to-service call)
   ▼
Cloud Run service "cost-advisor"  ← the ADK agent, its own service account
   ├── Gemini 3.8 Flash (Vertex AI)
   └── BigQuery (sample product catalog)

The page shows what ADK's own developer UI (adk web) shows: the agent graph, which agent is working, every tool call with its arguments and result, the transfers between agents in order, and what each agent saves in the session state.

The challenges, and how we solved them

1. "429 RESOURCE_EXHAUSTED" on the first real run

The first full run stopped halfway with:

429 RESOURCE_EXHAUSTED. Resource exhausted. Please try again later.

This is not a bug and not a billing problem. On pay-as-you-go, your project shares Google's Gemini capacity with other customers. When a model is busy, some requests are refused even if you sent very few. A multi-agent run makes many model calls in a short time, so it hits this much more often than a single chat call does.

What we did: we moved to a newer model (the recipe used gemini-3.5-flash; we use gemini-3.8-flash, the same model as the rest of our product) and added automatic retries with exponential backoff. In ADK this is one setting on the model object, and every agent that shares the model gets it:

from google.adk.models import Gemini
from google.genai import types

MODEL = Gemini(
    model="gemini-3.8-flash",
    retry_options=types.HttpRetryOptions(
        attempts=3,
        initial_delay=2,   # 2 s, then 4 s
        exp_base=2,
        http_status_codes=[429, 503],  # only "busy" and "unavailable"
    ),
)

Only 429 and 503 are retried. A real error, such as a bad request, still fails at once, so retries cannot hide bugs.

2. The microphone button and "not supported in the live api"

While testing in adk web, a click on the microphone produced:

1007 None. gemini-3.5-flash is not supported in the live api.

The microphone and camera buttons do not send a normal request. They open a two-way streaming connection to Gemini's Live API, which only works with Live-capable models. A normal text model is refused. The lesson: ADK runs an agent in two different ways, request/response (/run) and live streaming (run_live), and they need different models and a different design. We stayed with request/response.

3. "Tool 'wait' not found"

Halfway through the browser agent, the run failed with:

ValueError: Tool 'wait' not found.
Available tools: ... navigate, open_web_browser, ..., wait_5_seconds

The model asked for a tool called wait, and the agent only had wait_5_seconds. My first fix, an alias added before the model call, did not work. Reading ADK's source explained why: ADK's own Gemini model class renames the Computer Use action wait to wait_5_seconds (the name older Computer Use models knew) just before each request is sent, and deletes wait. That happens after the before_model_callback, so there was nothing to alias yet. Newer Gemini models still answer with wait.

What we did: fix the model's answer instead of the request, in an after_model_callback, which runs when the answer comes back and before ADK looks the tool up:

async def rename_wait_call_callback(callback_context, llm_response):
    """Points Gemini 3.x's `wait` call at `wait_5_seconds`."""
    if not llm_response.content or not llm_response.content.parts:
        return
    for part in llm_response.content.parts:
        if part.function_call and part.function_call.name == "wait":
            part.function_call.name = "wait_5_seconds"
            part.function_call.args = {}

Two lessons. First, a model can only call the tools you registered: if it asks for anything else, ADK refuses, which is a safety feature. Second, tool names are a contract with the model, and when models change, names drift. Agent evaluations (tests of whole runs) are how you catch that before your users do.

ADK gives you hooks at every stage of an agent's turn, and they are worth knowing early:

4. A call that took three minutes

One run looked frozen. The server log showed a single Gemini call that took 3 minutes and 3 seconds. Nothing was wrong; the page simply had nothing to show while it waited.

So I measured. Over two full runs there were 30 Gemini calls: the median took 5.1 seconds, the fastest 2.2 seconds, and 5 calls took more than 20 seconds, the slowest 60 seconds. One run of this recipe makes 15 to 26 calls, one after the other. A full run took between about 1 and 4 minutes.

Paying more is not the main answer. Reserved capacity (Provisioned Throughput) removes most of the slow outliers and the 429s, but it does not make 20 sequential calls fast. The time comes from the design:

• Too many model calls. Every hand-off between agents (transfer_to_agent) is a full model call. The comparison part alone goes coordinator → generator → coordinator → critic → coordinator.
• Thinking. Gemini 3 models reason before answering, and the recipe does not limit it, even for "pass this to the next agent".
• Long answers. Long Markdown reports take time to generate.
• A growing context. Every call resends the conversation so far. ADK even warns about it in the log ("no context_cache_config").

5. An agent that "forgot" its result

In ADK an agent saves its final answer to the session state under its output_key, and the next agent reads it from there. After one run the state had only 2 of the 5 expected values. The generator had handed control to another agent without writing its proposal first, and an agent saves its output_key only when it finishes with text.

Letting the model decide who goes next is flexible but unpredictable. When the order is fixed ("generate, then critique"), ADK's workflow agents (SequentialAgent, LoopAgent, ParallelAgent) run sub-agents in a guaranteed order, with no model call needed to decide it.

6. A run longer than the web request

Our API runs on App Engine with a 120-second request timeout. An agent run takes minutes, so we could not simply wait for it.

What we did: the agent service got two small endpoints next to ADK's own API. POST /runs creates a session, starts the run as a background task and answers at once; GET /runs/{id} returns the events so far. The page asks every 2 seconds and draws each new step as it arrives, which also feels much faster than a frozen screen. Because the run continues after the request has been answered, the Cloud Run service needs CPU that is always allocated (--no-cpu-throttling), and because the sessions live in memory, it runs as a single instance.

7. Keeping the agent private

The agent can spend money (every run is a set of Gemini calls), so it must not be reachable from the internet. We used the same pattern as our other internal services:

• the Cloud Run service has no public access; a call without a Google identity gets 403;
• only the App Engine service account may invoke it, and the backend sends a Google ID token with each call;
• the agent runs as its own service account with only what it needs: Vertex AI User, BigQuery Job User, and read access to one dataset.

One surprise: granting the dataset role with bq add-iam-policy-binding failed with "This feature requires allowlisting". The dataset's standard access list gives the same read permission, so we used that.

8. The cold start

The first time I opened the page in production, it showed an error. Cloud Run's log said:

The request was aborted because there was no available instance.

The service scales to zero when unused (so it costs nothing at rest), and the first request arrived while a new instance was starting. That start took 39 seconds, partly because the container started with uv run, which re-synced and rebuilt the Python project on every start.

What we did: start the server directly from the prepared virtual environment, turn on Cloud Run's startup CPU boost, and make the backend patient: while Cloud Run answers "not ready" (500, 502, 503), it asks again for up to about 40 seconds, and the page says the agent is waking up. Most of the remaining start time is loading ADK and its libraries, which is one more reason to keep agents small. Keeping one instance always warm would remove cold starts entirely, but with CPU always allocated that costs real money every month; it is a decision for production traffic, not for a lab.

What to consider before you build an agent

Design for few model calls. Count the calls one request will make. Read data with code and give the model the result, instead of letting it decide to call a tool for a step that never changes. Use workflow agents for fixed sequences and run independent work in parallel.

Choose the model and the thinking level per agent. A small, fast model with little thinking is enough to route or extract; save the strongest model for the step that needs reasoning.

Decide the experience by the task. A chat answer should come in seconds. A deep analysis can take a minute or more if it runs as a background job and shows its progress.

Plan for quotas. Retries with backoff from day one; reserved capacity once you have real traffic.

Decide where sessions live. In memory is fine for a lab, and nothing is stored. Production needs a managed session store, which the platform's managed runtime provides (called Agent Engine in older documentation and Agent Runtime in the 2026 announcements).

Treat tools as an attack surface. The recipe's own README warns that a browser-driving agent turns any URL the model writes into a request from your network. Its URL checks are defence in depth, not a boundary; the real control is network egress rules. Give each agent its own service account with the least access.

Test whole runs, not only functions. ADK has evaluation sets (adk eval). The wait problem above is exactly the kind of drift they catch.

Measure. Log how long each model call and tool takes. You cannot speed up what you have not measured.

Expect names to change. In April 2026 Vertex AI became the Gemini Enterprise Agent Platform, and documentation, blog posts and examples now use old and new names side by side. Check the current documentation for every step.

What comes next

Next, this example becomes a Cost Advisor: an agent that reads a cloud bill and the official price lists and proposes savings. It uses the lessons above: data read by code, specialists working in parallel, one critic pass, and small models where they are enough. The goal is a full cost report in about 30 seconds instead of 4 minutes, measured in the same Agent Lab.