Building agents with Gemini 2.0

An agent that plans, calls tools, critiques its own output, and retries when a step fails can easily fire ten or twenty model calls for a single user prompt. On expensive frontier endpoints, that retry loop blows through your token budget before the agent finishes the job.

At $0.15 per million input tokens and $0.60 per million output tokens, or roughly 20 cents per million tokens blended, Gemini 2.0 runs about 500 times cheaper than GPT-4.5 and 30 times cheaper than Claude 3.7. That pricing headroom makes multi-step reasoning chains, simulation-based planning, specialist sub-agents, and self-critique loops practical in production.

Set up Google AI Studio and the SDK

Sign in to Google AI Studio, accept the terms of service, and generate a new key in the API Keys section. Store the key as API_KEY inside a .env file in your project root, then install the @google/generative-ai package:

# Using npm
npm install @google/generative-ai

# Using yarn
yarn add @google/generative-ai

# Using pnpm
pnpm add @google/generative-ai

Lock agent output to a JSON schema

Left in free-text mode, a model will eventually wrap its tool call in markdown fences or prepend conversational filler that breaks your dispatcher. Passing a responseSchema inside generationConfig forces every turn to return a validated JSON object with thoughts, action, and actionInput:

import { GoogleGenerativeAI } from '@google/generative-ai';

const genAI = new GoogleGenerativeAI(process.env.API_KEY);

async function createAgentWithJSONMode() {
  const model = genAI.getGenerativeModel({
    model: 'gemini-pro',
    generationConfig: {
      responseSchema: {
        type: 'object',
        properties: {
          thoughts: {
            type: 'string',
            description: 'Agent reasoning process'
          },
          action: {
            type: 'string',
            description: 'Next action to take'
          },
          actionInput: {
            type: 'object',
            description: 'Parameters for the action'
          }
        },
        required: ['thoughts', 'action', 'actionInput']
      }
    }
  });

  // Your agent prompt and implementation here
}

When a task only needs a formatted prose report without downstream tool routing, plain numbered instructions in the prompt work without a schema:

import { GoogleGenerativeAI } from '@google/generative-ai';

const genAI = new GoogleGenerativeAI(process.env.API_KEY);

async function simpleAgentPrompting() {
  const model = genAI.getGenerativeModel({ model: 'gemini-pro' });
  
  const prompt = `
    You are a research assistant agent. I need you to:
    1. Find information about quantum computing advances in 2023
    2. Summarize the key breakthroughs
    3. Suggest three areas where these advances might be applied
    
    Format your response clearly with sections for each task.
  `;
  
  const result = await model.generateContent(prompt);
  console.log(result.response.text());
}

Offload math and verification to remote code execution

Language models routinely hallucinate arithmetic and prime factorizations when predicting tokens directly. With remote code execution, the agent writes and runs Python on the fly to compute the exact answer and verify its work before replying:

import { GoogleGenerativeAI } from '@google/generative-ai';

const genAI = new GoogleGenerativeAI(process.env.API_KEY);

async function runCodeExecution() {
  const model = genAI.getGenerativeModel({ model: 'gemini-pro' });
  
  // The agent can execute code to solve problems
  const result = await model.generateContent(`
    Write and execute a Python function that finds the prime factors of 294.
    Then tell me which of those factors is largest.
  `);
  
  console.log(result.response.text());
}

runCodeExecution();

Feed charts and screenshots as inlineData

When an agent needs to read a chart, diagram, document scan, or UI screenshot, read the image file into a buffer and pass it alongside the prompt as a base64 inlineData part on gemini-pro-vision:

import { GoogleGenerativeAI } from '@google/generative-ai';

const genAI = new GoogleGenerativeAI(process.env.API_KEY);

async function processImage() {
  const model = genAI.getGenerativeModel({ model: 'gemini-pro-vision' });
  
  // The image file you want to analyze
  const fileBuffer = readFileSync('path/to/image.jpg');
  
  // Convert the file to a GoogleGenerativeAI.Part
  const imagePart = {
    inlineData: {
      data: fileBuffer.toString('base64'),
      mimeType: 'image/jpeg',
    },
  };
  
  const result = await model.generateContent([
    'Describe what you see in this image and identify any relevant details.',
    imagePart,
  ]);
  
  console.log(result.response.text());
}

Configure safety thresholds per harm category

Default safety filters can block legitimate agent runs when summarizing user-generated content, security logs, or moderation queues. Pass explicit safetySettings with HarmCategory and HarmBlockThreshold values to match your application's risk profile:

import { GoogleGenerativeAI, HarmCategory, HarmBlockThreshold } from '@google/generative-ai';

const genAI = new GoogleGenerativeAI(process.env.API_KEY);

async function configureAgentSafety() {
  const model = genAI.getGenerativeModel({
    model: 'gemini-pro',
    safetySettings: [
      {
        category: HarmCategory.HARM_CATEGORY_HARASSMENT,
        threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,
      },
      {
        category: HarmCategory.HARM_CATEGORY_HATE_SPEECH,
        threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH,
      },
      {
        category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
        threshold: HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,
      },
      {
        category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
        threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,
      },
    ],
  });
  
  // Your agent implementation with custom safety settings
}