Introduction

In production, agent systems break in surprising ways. The LLM hallucinates a tool name. An API returns a 500. The user asks something outside the agent's scope. A loop runs forever because the model keeps calling the same tool with the same input. None of these are theoretical — every agent that runs for any length of time will hit them.

This lesson covers the patterns that make agents reliable: retries with backoff, input validation before tool calls, output validation after model responses, circuit breakers to stop runaway loops, and graceful fallbacks when the model produces something you cannot parse. The goal is not to make the agent perfect — it is to make failures predictable, logged, and recoverable.

Validating Inputs Before Tool Execution

Before you pass anything to an external tool, validate it. The most common failure mode is the agent generating a tool argument that looks syntactically right but is semantically nonsense — a date in the year 9999, a user ID that does not exist, a SQL query with a missing clause.

def get_user_orders(user_id: str, date_from: str, date_to: str) -> dict:
    # Validate user_id is numeric and exists
    if not user_id.isdigit() or not user_exists(user_id):
        return {"error": "invalid_user_id", "message": f"User {user_id} not found"}

    # Validate date format
    try:
        from datetime import datetime
        datetime.strptime(date_from, "%Y-%m-%d")
        datetime.strptime(date_to, "%Y-%m-%d")
    except ValueError:
        return {"error": "invalid_date", "message": "Dates must be YYYY-MM-DD"}

    # Validate date range
    if date_from > date_to:
        return {"error": "invalid_range", "message": "date_from must be before date_to"}

    # Now safe to call the actual tool
    return query_orders(user_id, date_from, date_to)

When validation fails, return a structured error object — not an exception. The agent reads this and decides what to do next: ask the user for clarification, try a different approach, or give up gracefully.

Retries, Backoff, and Circuit Breakers

Transient failures (network timeouts, 503 errors, rate limits) deserve retries. But naive retries make things worse — if the downstream service is overloaded, hammering it harder just keeps it down.

import time
import random

def call_with_retry(fn, max_attempts=3, base_delay=1.0):
    for attempt in range(max_attempts):
        try:
            result = fn()
            if attempt > 0:
                log_recovery(attempt)  # Log that retry succeeded
            return result
        except RetryableError as e:
            if attempt == max_attempts - 1:
                raise
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
            log_retry_attempt(attempt, e, delay)
            time.sleep(delay)

Exponential backoff with jitter spreads retry traffic so you do not synchronize all your agents retrying at the same instant. Only retry on errors classified as transient — never retry on 400 Bad Request, because the same input will produce the same failure every time.

Circuit breakers go one step further: if a tool has failed N times in a row, stop calling it for a cooldown period and fail fast. This protects you from agents looping into a dead service.

class CircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown_seconds=60):
        self.failures = 0
        self.threshold = failure_threshold
        self.cooldown = cooldown_seconds
        self.open_until = 0

    def call(self, fn):
        if time.time() < self.open_until:
            raise CircuitOpenError(f"Circuit open until {self.open_until}")
        try:
            result = fn()
            self.failures = 0
            return result
        except Exception:
            self.failures += 1
            if self.failures >= self.threshold:
                self.open_until = time.time() + self.cooldown
            raise

Validating Model Output and Handling Loops

The model will sometimes produce output that does not match your expected schema — a JSON blob missing a required field, a tool call with arguments in the wrong type, a "function call" that is actually a hallucinated natural-language sentence. You need a layer that catches this.

def parse_agent_response(raw: str) -> AgentAction:
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError:
        return AgentAction.recover("Model returned non-JSON output")

    if "tool" not in parsed or "arguments" not in parsed:
        return AgentAction.recover("Missing tool or arguments field")

    if parsed["tool"] not in ALLOWED_TOOLS:
        return AgentAction.reject(f"Model tried to call unknown tool: {parsed['tool']}")

    return AgentAction.execute(parsed["tool"], parsed["arguments"])

Loop detection is equally important. Track the last N (tool, arguments) pairs. If the same pair repeats more than M times, abort the agent and surface an error to the user — something like "I seem to be stuck repeating the same step." Without this guard, a confused model can burn thousands of tokens calling the same dead-end tool forever.

Graceful Degradation and User-Facing Errors

When the agent cannot complete a task, the response to the user should be specific and actionable. "An error occurred" is useless. "I could not retrieve your orders because the order service is temporarily unavailable. Please try again in a minute, or I can show you your account summary instead" tells the user what happened and offers an alternative.

Build a fallback chain: if the primary tool fails, try a simpler secondary tool (for example, cached data instead of live data, or a different LLM call with a smaller scope). If everything fails, return a final response that explains the limitation honestly. Never silently make up data to cover a missing tool call — users catch this and trust is gone.

Summary

Reliable agents are built from layers of validation, retry, and graceful failure handling. Validate inputs before tools, retry transient errors with exponential backoff and jitter, use circuit breakers to stop runaway loops, validate model output against a strict schema, detect repeated actions to prevent infinite loops, and always give the user a specific, honest answer when things go wrong. The patterns in this lesson apply whether you are building with LangChain, custom orchestration, or any other agent framework — the framework may handle some of these for you, but you should still understand what is happening underneath.