how to deploy AI agents in production
Deploying an AI agent in production means containerizing the logic, wiring it to models and data, and running it on infrastructure that scales. The hard part is managing state, observability, and guardrails so the agent behaves when running without supervision.
Last updated
Deploying an AI agent in production means containerizing the logic, wiring it to models and data, and running it on infrastructure that scales. The hard part is managing state, observability, and guardrails so the agent behaves when running without supervision. A production deployment is less about the model choice and more about the engineering around it.
How do you package an AI agent for deployment?
You package an agent the same way you package any service, by putting its code, dependencies, and runtime into a container. The container needs to know how to reach the language model via API keys or local endpoints and how to reach any tools the agent uses. Those tools might be databases, internal APIs, or external services. The environment variables and secrets must be injected at runtime rather than baked into the image.
A good agent package is stateless at the container level. The container should be able to die and restart without losing the context of the user it was serving. This means pushing conversation history and intermediate steps out to a database like Postgres or Redis before the agent responds. When the container restarts, it fetches state from the database and continues.
Where does the agent actually run?
An agent in production runs on a compute platform that can restart failed processes and scale horizontally under load. Kubernetes is the standard choice because it handles pod scheduling, health checks, and rolling updates. You define a deployment that specifies how many replicas of the agent container you want and Kubernetes keeps that many running at all times.
If you run on the cloud, you use managed Kubernetes or a serverless container service. If you run on your own hardware, you join your nodes to the cluster and Kubernetes schedules the agent pods there. The platform matters less than the guarantees it provides. You need automated restarts, traffic routing, and the ability to roll back a bad deployment without manual intervention.
How do you handle state and memory?
Agents need memory to carry on a conversation and to remember facts about the user. Short term memory covers the current conversation and usually lives in a fast cache or database table keyed by a session id. Long term memory covers user preferences and historical interactions and usually lives in a vector database or a structured store.
State management becomes a distributed systems problem when you have multiple agent replicas. If a request from the same session hits a different replica, that replica must be able to load the same state. You cannot rely on local memory inside the agent container. Every replica must read from and write to the same external state store.
How do you monitor an agent in production?
Standard infrastructure metrics like CPU and memory usage tell you if the agent is up but not if it is doing the right thing. You need observability tools that capture the prompts going in and the completions coming out. This usually means logging the full payload of every model call to a tracing system so you can inspect why an agent made a certain decision.
You also need guardrails. A guardrail is a check that runs before or after the agent takes an action. It can validate outputs, restrict which tools the agent is allowed to call in a given context, or block certain topics entirely. Without guardrails and tracing, a production agent is a black box that can hallucinate or take destructive actions with no record of why.
What my-app.engineer does about this
On my-app.engineer you describe an application in chat and the platform builds and deploys it into a Kubernetes cluster you own. The agents that build and run your app operate inside that environment, using the cluster tools you add from the catalogue for state, tracing, and browser access. You get the production infrastructure without writing the deployment files yourself.