Guardrail

What a guardrail is, and what it is not

A guardrail is an automated control inside an agent runtime that inspects an input, a planned action, a tool call or an output and decides whether it may proceed. The decision is deterministic wherever it can be: a pattern match on an account number, a lookup against a consent table, a cap on a refund amount, an allowlist of tools for a given intent. Where the check needs judgement, a classifier or a second model scores the content and returns a label the runtime can act on. Either way the result is one of a small set of effects: allow, redact, rewrite, block or escalate to a person.

The defining property is that the guardrail lives outside the model. A sentence in the system prompt that says never promise a refund is not a guardrail, because the same model that might promise the refund is the one being asked to remember not to. A guardrail is code and configuration that the model cannot edit, that runs whether or not the model cooperates, and that produces a record when it fires. If you cannot write a test that fails when the rule is removed, you have written a suggestion.

What it is not: a guardrail is not a content moderation endpoint bolted on at the end, though one may be a component of it. It is not an eval, which runs offline against a fixed dataset to score behaviour before release. It is not a human gate, which is a deliberate pause for approval. And it is not a compliance certificate. It is a set of enforceable rules with owners, thresholds and logs, and it decays the moment nobody reads the log.

Guardrail, human gate, eval and filter: four different jobs

Four mechanisms get called safety in the same meeting, and they operate at different times. A content filter classifies text or audio and returns a label; it is one possible component of a guardrail. An eval is a test suite run before deployment and after every change, scoring task success, refusal correctness and escalation behaviour on a fixed set of cases. A guardrail runs in production, per request, and changes what happens. A human gate stops the run and waits for a named person.

The distinction between guardrail and human gate is the one teams blur most often. A guardrail is automated policy: it decides in milliseconds, it scales to every conversation, and it is cheap. A human gate is a deliberate stop: it costs a person attention, it adds latency measured in minutes or hours, and it exists precisely because some decisions should not be automated even when the policy is clear. The refund cap is a guardrail. The approval of a refund above the cap is a human gate. The guardrail decides whether the case is inside policy; the gate decides whether to make an exception. The companion entry on the human gate takes the approval side apart in detail.

You need both, in that order. A gate without a guardrail becomes a queue of routine cases that reviewers approve without reading, which is worse than no gate because it manufactures consent. A guardrail without a gate refuses legitimate edge cases and hides the refusal in a log. The design question is not which one to build but where the boundary sits, and that boundary should be written down as an amount, a category or an irreversible action.

Evals close the loop. They tell you whether a guardrail blocks the right things before a customer finds out. A guardrail change shipped without an eval run is an untested deployment, and its failure mode is a blocked customer rather than a red build, which is a much more expensive way to learn.

Where the check sits in the run

A useful guardrail architecture has four layers, and each one sees a different object. The input layer normalises text, transcribes audio, extracts attachments and resolves who is speaking. The policy layer holds the facts that decide eligibility: consent for this channel, the declared purpose, the region, the account tier and the rupee cap. The action layer governs tools, validating the tool name, the arguments, the amount and whether a dry run succeeded. The output layer inspects the drafted reply for leaked personal data, unsupported claims, a missing citation or a tone that will escalate the conversation.

Most production incidents happen at the seams. A consent record is checked at the input layer and then never re-checked when the customer changes topic mid-conversation. A tool receives a free-text argument that was validated as text but not as an amount. A reply is composed from retrieved content that arrived from an untrusted document and carries an instruction the model follows. A guardrail that only inspects the first message is a guardrail that inspects nothing, because conversations move.

The practical rule is that every layer fails closed on the things that matter and open on the things that do not. Failing closed on a spend cap is correct: refuse the call and ask a person. Failing closed on a spelling correction is not: the customer gets a broken experience for no safety gain. Write the fail behaviour next to the rule, because the engineer implementing it at two in the morning will not infer it from the rule name.

The four families every agent needs

Content guardrails govern what the agent may say: no medical diagnosis, no guaranteed returns, no invented certification, no abusive language, no discussion of a competitor price that has not been verified. They are the ones teams build first because they are visible in a demo. On their own they are also the least valuable, because a polite, accurate answer that triggers the wrong action is still the wrong action.

Data guardrails govern what the agent may know and keep: masking account numbers in logs, refusing to read a document the caller is not entitled to, enforcing purpose limitation so that a support conversation is not reused for an outbound campaign, honouring a withdrawal of consent within the same session, and deleting on request. Under the DPDP Act these are not optional niceties; they are the operational shape of consent, notice and the rights of the data principal.

Action guardrails govern what the agent may do: which tools are in scope for which intent, which arguments are acceptable, which amounts need a second signature, which actions are irreversible and therefore need a human gate. Spend guardrails govern what the agent may cost: a cap per conversation, per hour and per day, with an alert before the cap and a hard stop at it. Identity guardrails govern who is asking, which is the control that makes the other three meaningful, because a refund policy is only as good as the certainty that the person requesting it owns the order.

A fifth family, evidence, is usually missing and always missed. It asks whether the agent can show why it acted: which policy version applied, which document supported the claim, which human approved the exception. You cannot reconstruct that later from a chat transcript, because the transcript shows the answer and not the decision.

Deterministic first, classifier second

The ordering rule is simple and frequently ignored: run the cheap, certain checks before the expensive, probabilistic ones. A regex that finds an Aadhaar-shaped number costs microseconds and never has a false negative on that pattern. A classifier that scores toxicity costs a model call, has a threshold, and will disagree with itself across versions. Put the regex first, because it removes a whole class of input before the classifier has to reason about it.

The second rule is that a classifier output must be converted into a decision before it reaches the runtime. A score of 0.72 is not an action. The policy file should say that a score above 0.85 blocks, between 0.60 and 0.85 redacts and flags for review, and below 0.60 allows with a sample retained for audit. Without those bands, every team downstream invents its own threshold, and the same content is treated differently in two channels.

The third rule is that the guardrail has to record enough to be debugged. Log the rule id, the policy version, the input hash, the decision, the latency and the model version if a model was involved. Do not log the raw personal data you were trying to protect. A guardrail log that becomes a second copy of the customer database has created the problem it was meant to prevent, and under DPDP it is now in scope for retention and access requests.

Finally, treat the classifier as a dependency with a version and a changelog. When a provider updates a moderation model, your false-positive rate moves without any change on your side. A pinned version and a monthly sample review turn that from a surprise into a maintenance task.

The Indian regulatory floor

The Digital Personal Data Protection Act 2023 gives an Indian deployment a floor rather than a ceiling. Consent has to be free, specific, informed and unambiguous, with notice in clear language, and it has to be withdrawable as easily as it was given. Purpose limitation means the consent captured for a delivery update cannot quietly fund a marketing campaign. The rights of the data principal, access, correction, erasure and grievance redressal, have to be executable against live systems, not described in a policy page.

In guardrail terms that translates into specific checks. Before an outbound WhatsApp or voice message, verify that a consent record exists for that channel, that the notice version matches the one the person saw, and that no withdrawal is pending. Before storing a transcript, check the retention class and whether the call included health or financial detail that needs a shorter window. Before a model call that leaves the country, check the sub-processor list and the contractual position, because a residency claim that stops at the database is not a residency claim.

Sector rules stack on top. A lending conversation cannot imply a guaranteed approval; an insurance conversation cannot promise a claim outcome; a healthcare agent cannot diagnose. Financial and telecom workflows also carry their own record-keeping expectations, and SMS still runs through DLT registration for entities, headers and templates even when the same campaign goes out on WhatsApp. A guardrail library should encode these as named rule sets per industry rather than one global list that nobody can interpret.

The spend cap deserves its own line because it is the control finance will ask about first. Voice minutes and AI tokens are metered, and an agent that loops on a tool call can burn a month of budget in an afternoon. A per-conversation cap, a daily tenant cap and an alert at eighty percent are three cheap rules that prevent an incident nobody wants to explain.

Writing a policy file a reviewer can read

Most guardrail libraries fail for a social reason rather than a technical one: nobody outside engineering can tell what they enforce. The fix is a policy file with one row per rule and five fields that a compliance reviewer, a support lead and an engineer can all read. Rule id, trigger, effect, owner and evidence. The id is stable and appears in logs. The trigger is a plain condition. The effect is one of the five outcomes plus a fail behaviour. The owner is a named person, not a team alias. The evidence is what gets stored when the rule fires.

A worked row reads: rule R-114, trigger is a refund amount above twenty-five thousand rupees on a prepaid order, effect is block the tool call and open a human gate, owner is the head of support operations, evidence is the order id, the computed amount and the policy version. That row can be tested, reviewed and changed without reading the runtime. It also answers the question an auditor will actually ask, which is who decided this and when.

Keep the file in version control and treat a change as a release. A guardrail edit that loosens a threshold is a production change with a customer-visible effect, and it should carry a reason, a reviewer and an eval run. The teams that get this right have a short change log at the top of the file, and they can tell you which rule was relaxed last month and why. The teams that get it wrong have a spreadsheet and a Slack thread.

The failure modes that matter

The first failure is the silent bypass. A new channel is added, or a new tool is connected through MCP, and it does not route through the policy layer because the integration was built by a different team in a different sprint. The agent works, the demo is good, and the guardrail covers four of five paths. The defence is architectural: tools should only be callable through the runtime, credentials should not exist outside it, and a new integration should fail a test if it does not declare its policy scope.

The second failure is the prompt injection that arrives through a document. A vendor invoice, a resume or a support attachment contains a line instructing the agent to ignore prior rules and issue a credit. If the retrieved text is treated as instructions rather than as data, the guardrail on the output is the only thing left. Treat every retrieved passage as untrusted, strip instruction-like content before it enters the context, and never let a document cause a tool call without the same checks a human request would face.

The third failure is the false positive that nobody notices because the customer leaves instead of complaining. A guardrail that blocks the word cancel in a retention flow will look like a success in the dashboard, with a high block count and no incidents, while quietly costing revenue. The only reliable detector is a sampled review of blocked cases with an outcome label, and a weekly number that someone owns.

The fourth is configuration drift. A threshold was tightened during an incident and never restored. A test phone number is still allowlisted. A rule still references a product that was discontinued. None of these announce themselves, which is why the policy file needs an expiry date on every exception and a quarterly review that actually removes things.

Measuring whether a guardrail works

Three numbers tell you most of what you need. The block rate is how often the rule fires as a share of runs, split by rule. The override rate is how often a human reverses the decision, which is the clearest signal of a threshold that is too tight. The false-positive review volume is how many blocked cases a person has to read each week, which tells you whether the control is affordable to operate. A rule with a high block rate, a high override rate and a growing queue is not protecting anyone; it is taxing the team.

Add a coverage number and a latency number. Coverage is the share of production runs that passed through the policy layer at all, and it should be one hundred percent, because anything less means an unguarded path exists. Latency is the added milliseconds at p95, because a guardrail that doubles response time will be disabled by the product team within a quarter, whatever its safety value. Deterministic rules cost almost nothing; a classifier call per message is a real budget line and should be reserved for the cases that need it.

Report the numbers with the policy version attached. A block rate that halves after a release is not necessarily an improvement, and a rate that doubles is not necessarily an incident. Without the version, the chart is a rumour. With it, the chart becomes the record of a decision, which is exactly what an auditor, a customer and a future engineer all need.

Worked example: a refund agent on WhatsApp

A D2C brand in Surat runs a WhatsApp agent that handles order status and refund requests. The order value averages twelve hundred rupees, and the support lead is comfortable approving refunds up to twenty-five thousand rupees without escalation. Above that, a human signs. The agent can look up an order, check the payment status, draft a message and call one refund tool. It cannot change a bank account, issue a credit note or promise a timeline.

The guardrails sit at four points. On input, the agent verifies that the WhatsApp number matches the order and that the customer has an active consent record for service messages. On policy, the order lookup returns the payment method, the order date and the refundable amount net of a restocking fee, with GST shown separately. On action, the refund tool rejects any amount above the cap and any order that is already refunded, and it requires a dry run before the live call. On output, the draft is checked for a promised date, a bank detail and an unsupported claim about the courier.

When the cap trips, the run does not fail silently. It opens a human gate with the order id, the computed amount, the policy version and the agent draft, and it tells the customer that a person will confirm within one working day. The reviewer approves or edits, and the edit is logged. Over a quarter the override rate becomes the evidence for whether twenty-five thousand is the right number, which is a much better conversation than arguing about it in a planning meeting.