Why production agents need pinned prompts and tool versions
If the prompt, model, or tool schema can change without a ticket, you do not have a production system. You have a live experiment sitting on customer work.
This week's headlines roundup flagged version pinning for prompts, models, and tool schemas. That is the right fight. Last month we talked about where code runs. This week is about what actually ran when a customer asked a question.
I still see teams edit a system prompt in a vendor console on a Thursday, then spend Friday arguing about why invoices look different. Nobody can prove which instructions the agent had. That is not an AI problem. That is a change-control problem.
What to pin (and what not to)
Pin the pieces that change behavior:
- Model ID, including a date-stamped snapshot if the vendor offers one.
- Prompt text stored as a file or record with a hash, not a sticky note.
- Tool and MCP schemas so a renamed field cannot silently break writes.
- Retrieval corpus version or index build ID if answers depend on your docs.
Do not pin the customer's question. Do not pin live telemetry. Those belong in traces. Pin the policy the agent used, then record the request next to it.
Questions I ask in a review
- Can you show me the exact prompt that served this ticket last Tuesday?
- Who can edit production prompts, and is that change in git or a ticket?
- If the vendor ships a new model alias, do you stay on the old ID?
- When a tool schema changes, does the agent fail closed or improvise?
- How do you roll back one agent without rolling back the whole stack?
If the answers are vague, you are not ready for production evals either. Evals against a moving prompt are theater.
A simple promotion path
Steal the path you already use for websites:
Draft in a channel or staging tenant.
Eval against a fixed set of tickets or invoices.
Approve through the same
human queue
you use for money moves.
Promote a version number into production.
Watch traces and a small eval set for 48 hours.
Hotfixes still happen. They should be a new version with a note, not a silent overwrite. Pair this with your incident playbook so a bad prompt has a rollback owner.
Where teams get sloppy
- Shared prompts across products so a marketing tweak hits finance.
- Vendor "latest" aliases that change overnight.
- Plugin updates that ship new tool fields without a schema bump.
- No hash in the log, so you cannot prove what ran.
Those gaps sit next to ungoverned MCP tools and unowned agent identities. Version pinning will not save a tool that should never have been connected.
How we run this at Swift Media
Jarvis work stays in scoped channels with a paper trail. When we change how an agent writes a page or a ledger, we treat it like a deploy: named version, someone who can roll it back, and a log you can open later. If you want help putting one production agent on that path, talk with us.