Building Production Agents

What you'll learn in this module

• How to make agents reliable with retries, fallbacks, and graceful degradation
• How to control and optimize costs in production
• How to test agents systematically before and after deployment
• How to deploy with observability and monitoring

Reliability Engineering

Production agents fail. External APIs go down, LLMs hallucinate, rate limits hit. Reliability engineering is about designing for these failures.

The reliability hierarchy

Retry strategies

Not all failures are equal. Match your retry strategy to the failure type:

Exponential backoff implementation

async function withRetry<T>(
  fn: () => Promise<T>,
  maxRetries = 3,
  baseDelay = 1000,
): Promise<T> {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (error) {
      if (attempt === maxRetries) throw error;

      const isRetryable =
        error.status === 429 || error.status === 503 || error.code === "TIMEOUT";

      if (!isRetryable) throw error;

      // Exponential backoff with jitter
      const delay = baseDelay * Math.pow(2, attempt) + Math.random() * 1000;
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
  throw new Error("Unreachable");
}

Fallback chains

When the primary model is unavailable or too slow, fall back to alternatives:

Cost Optimization

AI agents can be expensive. A poorly designed agent might make 50 LLM calls to answer a simple question. Systematic cost optimization keeps budgets under control.

Cost drivers

The model routing pattern

Not every task needs the most powerful model. Route based on complexity:

Caching strategies

Cost monitoring

Track these metrics per agent, per day:

Testing Strategies

Agents are non-deterministic. You can't write a unit test that asserts exact output. Instead, use a layered testing approach.

Testing pyramid for agents

Unit tests

Test everything that doesn't involve an LLM call:

// Test tool adapters
test("listOpenIssues returns simplified format", async () => {
  const issues = await listOpenIssues("org/repo");
  expect(issues[0]).toHaveProperty("number");
  expect(issues[0]).toHaveProperty("title");
  expect(issues[0]).not.toHaveProperty("body"); // Should be stripped
});

// Test input validation
test("rejects invalid repo format", async () => {
  await expect(listOpenIssues("not-a-repo")).rejects.toThrow("owner/repo");
});

// Test output parsers
test("parseAgentResponse handles malformed JSON", () => {
  const result = parseAgentResponse("```json\n{broken\n```");
  expect(result.success).toBe(false);
  expect(result.error).toContain("JSON");
});

Evaluation tests

Build a test dataset that covers your agent's expected scenarios:

[
  {
    "input": "What are the open bugs for project-x?",
    "expected_tools": ["list_open_issues"],
    "expected_contains": ["bug", "project-x"],
    "max_tool_calls": 3,
    "max_latency_ms": 5000
  },
  {
    "input": "Summarize last week's standup notes",
    "expected_tools": ["search_documents", "summarize"],
    "expected_contains": ["standup", "summary"],
    "max_tool_calls": 5,
    "max_latency_ms": 10000
  }
]

Run the evaluation:

Regression testing

After every change to prompts, tools, or models:

1. Run the full evaluation test suite
2. Compare metrics against the previous baseline
3. Flag any regression > 5% in key metrics
4. Investigate before deploying

Observability

You can't fix what you can't see. Agent observability means capturing enough data to diagnose failures and optimize performance.

What to log

Tracing structure

Each span captures:

• Start/end timestamp
• Operation type (LLM call, tool call, evaluation)
• Token usage (for LLM calls)
• Input/output (optionally sampled for privacy)
• Error information (if any)

Dashboards

Build dashboards around these four views:

Deployment Patterns

Canary deployment

Roll out changes to a small percentage of traffic first:

Shadow mode

Run the new agent in parallel without serving its results to users:

1. Send every request to both v1 and v2
2. Serve v1's response to the user
3. Log and evaluate v2's response silently
4. Compare quality metrics before switching

Feature flags

Wrap agent capabilities behind feature flags:

async function handleTask(task: Task) {
  if (featureFlags.get("use_new_planner")) {
    return await newPlannerAgent(task);
  }
  return await currentAgent(task);
}

This lets you enable new capabilities for specific users, teams, or a percentage of traffic.

Production Checklist

Before deploying any agent to production:

Reliability

• Retry logic with exponential backoff for transient failures
• Fallback model or cached response for complete outages
• Iteration limits on all loops (max 10-15 iterations)
• Timeout on all external calls
• Input validation at system boundary

Cost

• Token budget per task
• Model routing (cheap model for simple tasks)
• Caching for repeated queries
• Daily spend alerts

Testing

• Unit tests for all tool adapters
• Evaluation dataset with 50+ test cases
• Quality baseline established
• Regression tests passing

Observability

• Execution tracing enabled
• Dashboards for health, cost, quality, throughput
• Alerts configured for error rate and cost spikes
• Log retention policy set

Security

• Tools have minimal permissions (principle of least privilege)
• User input is sanitized before inclusion in prompts
• Sensitive data redacted from logs
• Agent scope boundaries documented and enforced

What you now understand

Congratulations — you've completed the Agentic AI Workflows learning path. You now have the architecture patterns, implementation strategies, and production practices to build AI agent systems that are reliable, cost-effective, and observable.

Apply what you've learned:

• Build agentic workflows visually in Skytells Orchestrator
• Use the Skytells AI API to power your agents with state-of-the-art models
• Explore the Orchestrator Mastery path to learn the visual workflow builder in depth