
How to build an AI agent in six steps: define the job and its boundaries, give it tools not access, add the review step, build the evaluation set, run it next to the current process, then harden and monitor it in production. An AI agent is a model that is allowed to act: read a record, call a system, send a message, change a status. That is what makes agents useful and what makes most first attempts fail. The demo that books a meeting is easy. The agent that handles two hundred intake requests a day, inside a regulated business, without anyone discovering a month later what it did, is engineering. This is the process we use at Speer, written for the people who have to decide whether to build one.
Start with a task that has volume, a checkable right answer and a tolerable cost of error: triaging support requests, extracting fields from documents, drafting replies from your own policies, enriching tickets. Write down the inputs, the allowed outputs and the boundaries in one page. "May not send anything externally." "May not change a payment." "Escalates when confidence is low." If you cannot write that page, you are not ready to build; you are ready to run a pilot with a person doing the job while you watch what the decisions actually are.
An agent should act through a small set of tools you define, each with a narrow purpose and its own permissions: look up a customer, create a ticket, draft a message for review. Never hand it a database password or an admin account. Tools are where you enforce the boundaries from step one, log every action under a named service identity, and make the agent's behaviour testable. If a tool can do something irreversible, it asks a person first.
Every output that affects a person, a payment or a record of consequence goes through a review step until the evaluation data earns trust, and some steps stay human permanently by design. Design the review queue as a real interface: what the agent proposed, why, the evidence it used, one click to approve or correct. Corrections are gold; they become your test set.
Collect fifty real cases with known good outcomes. Write down how you will score an answer: exact match, rubric, or a checklist. Run every change to the prompt, the tools or the model against that set and watch accuracy, refusal rate, latency and cost per task. Grow the set with every failure you find in production. An agent without an evaluation set is a demo with a login page; you cannot know whether a change made it better or worse.
For most business agents, a current hosted model from a major provider, behind an interface you can swap, is the right starting point. Give it your own knowledge through retrieval over your documents and records rather than fine-tuning. Keep state in your systems, not in the model's context: the ticket, the case file, the audit log. Fine-tune only when you have proprietary data at scale or strict cost and latency targets, and only after retrieval has been tuned.
Pilot in one workflow, with real data, for four to eight weeks, while the existing process keeps running so the comparison is honest. Then production: rate limits, caching, fallbacks when the provider is slow, monitoring of output quality over time, cost controls per task, and a way to roll back a prompt change the way you roll back code. Write the governance document: what the agent does, what data it touches, what it must not do, who reviewed it and when.
Starting from the model instead of the job. Giving the agent broad access because it was faster than writing tools. No review step on decisions that affect people. Judging quality by the demo instead of a test set. Letting the agent answer from general knowledge when it should answer from your records. Treating prompts as code nobody owns, versions or tests. We run a practical workshop on exactly this, Build Your First AI Agent, and we build agents for partners through our AI automation services. The guide on how to build an AI app covers the product around the agent.
A model that can take actions through tools you give it, not just produce text: look up records, call systems, draft and send messages, change statuses. The permissions, review and logging around those actions are what separate a production agent from a chatbot.
A pilot in one workflow takes four to eight weeks. A production agent with evaluation, monitoring and governance typically lands three to five months after kickoff. Later agents reuse the first one's tools and controls and arrive faster.
Usually not. Retrieval over your own documents and records, good tool design and a solid evaluation set matter more than the model. Fine-tune when you have proprietary data at scale or strict cost and latency targets.
Pick the simplest thing your engineers can test: a thin loop around the model provider's tool-calling API, with your tools as plain functions. Frameworks help with plumbing but do not solve permissions, evaluation or review, which is where the work is.
Least-privilege tools under a named service identity, a human review step for consequential actions, an audit log of every action, production data kept in your environment, and an evaluation set that runs on every change.
Yes, and it should. Agents earn their keep by acting in the system of record, so the integration layer, through APIs to your EMR, CRM, ERP or ticketing system, is most of the build.












