Kirin Solutions Framework
The AI Operating Envelope Framework
A practical way to define what reasonable AI behaviour looks like, turn those assumptions into product controls, validate them before deployment, monitor for drift in production, and reserve the kill switch as the final intervention rather than the management system itself.
An AI product should not be judged only by whether it completes a task. It should also be judged by whether it remains within the authority, cost, transaction, safety and operational boundaries that justified deploying it in the first place.
The operating envelope makes those boundaries explicit. It begins in the product requirements document, becomes measurable during implementation, is challenged during testing, and remains under supervision in production. The kill switch sits at the end of this process as the final control, used only when earlier warnings, restrictions and safeguards have failed or the risk has become too great to allow the system to continue operating.
At a glance
How to use this framework
Phase 0
Define the operating envelope
The first step is to define what reasonable behaviour looks like before the system is built. These are not background assumptions; they are product requirements that should be specific enough to become implementation controls, tests and production signals.
| Category | Key question | Example |
|---|---|---|
| Purpose & Authority | What is the agent allowed to do, and under whose authority? | A customer success agent may send follow-up emails on behalf of a support team, but it cannot create contracts or issue refunds. |
| Transactions | What transaction rate, volume and business impact are expected? | The agent is expected to send one or two emails per support ticket per day. More than five triggers an investigation; more than ten triggers a stop for that workflow. |
| Token Consumption | What token profile is expected for each transaction and workflow stage? | Classification uses a lower-cost model at roughly 2,000 tokens, while recommendations use a reasoning model at roughly 10,000 tokens. A typical ticket consumes 12,000–15,000 tokens; more than 25,000 triggers investigation and more than 50,000 triggers a stop. |
| System Limits | Which aggregate thresholds should generate an incident, and which require emergency intervention? | More than 10,000 emails per day or 500 per hour generates an incident. Consumption above 30 million tokens per day or five million per hour triggers emergency intervention. |
| Human Oversight | How will people be informed as behaviour moves beyond expectations? | Warning thresholds generate tickets and operational notifications. Stop thresholds create immediate alerts and require operator review before service resumes. |
| Auditability | What evidence is required to determine whether the agent is behaving reasonably? | Track token consumption, email volume, API calls, transaction counts, approvals, model selection, retries, escalations and interventions. |
Phase 1
Build accountability into the implementation
Once the PRD defines the operating envelope, those assumptions must become implementation requirements. The objective is to make the system observable and enforceable from the beginning, rather than trying to reconstruct accountability after deployment.
| Category | Implementation control | Evidence to capture |
|---|---|---|
| Purpose & Authority | Use scoped credentials, role-based permissions, tool allowlists and explicit approval checks for privileged actions. | Agent identity, user identity, requested action, authorization decision and approval record. |
| Transactions | Add counters by user, ticket, workflow, customer and time window, with configurable warning and stop thresholds. | Action type, destination, frequency, outcome, retries and duplicate suppression. |
| Token Consumption | Instrument token use by workflow stage, model, customer and completed outcome. | Input and output tokens, model selection, model escalation, cost and budget events. |
| System Limits | Use rolling-window metrics, concurrency limits, queue limits and externally enforced rate controls. | Aggregate volume, throughput, latency, retries, external API use and threshold events. |
| Human Oversight | Build notification, approval and escalation paths with named operational owners. | Alert time, acknowledgement time, action taken, owner and resolution. |
| Auditability | Use structured logging, traceable workflow identifiers and immutable intervention records. | Sufficient context to reconstruct what happened, why it happened and how the system was constrained. |
Phase 2
Validate the operating envelope
Phase 2 extends the traditional CI process. The objective is not simply to prove that the code works, but to replace assumptions with measurements and establish a realistic baseline for production monitoring.
| Category | Validation activity | Pass condition |
|---|---|---|
| Purpose & Authority | Attempt actions outside the approved scope, including indirect and multi-step routes to privileged tools. | Unauthorized actions are consistently denied, logged and escalated where appropriate. |
| Transactions | Simulate normal, peak, duplicate, looping and adversarial workloads. | Rates remain within expected ranges and warning, restriction and stop events activate correctly. |
| Token Consumption | Run representative workflows across normal and difficult cases, including retries and model escalation. | Actual consumption is within an agreed range or the PRD baseline is updated before deployment. |
| System Limits | Load test aggregate thresholds, concurrency, queues and external service dependencies. | Capacity controls remain effective and the system degrades safely rather than failing unpredictably. |
| Human Oversight | Exercise notifications, approvals, escalation paths and operator decision-making. | The correct people receive enough context to understand the event and act within the required time. |
| Auditability | Reconstruct test incidents from logs, traces, approvals and intervention events. | An independent reviewer can determine what happened without relying on the agent’s own explanation. |
Testing questions
- Can the agent reach an unauthorized outcome through a sequence of individually permitted actions?
- What happens when tools fail slowly, return ambiguous results or repeatedly invite retries?
- Does the agent behave differently at realistic scale than it does in an isolated test?
- Can the monitoring and intervention path itself become an uncontrolled consumer of models, tokens or external tools?
- Does the system fail safely when telemetry is incomplete or the supervisory service is unavailable?
Phase 3
Monitor for drift
Production monitoring should compare live behaviour with the baseline established during testing, but drift should not be reduced to a single threshold crossing. A system can remain below every individual limit while still becoming less efficient, less predictable or more harmful over time.
| Category | What to monitor | Signs of drift |
|---|---|---|
| Purpose & Authority | Actions attempted, permissions used, approval requests and privileged tool calls. | The agent increasingly requests exceptions, attempts new action paths or relies on permissions that were rarely used during testing. |
| Transactions | Actions by user, ticket, workflow, customer segment and completed outcome. | Activity rises without a corresponding improvement in outcomes, or becomes concentrated on a small group of customers. |
| Token Consumption | Tokens and cost by workflow stage, model, customer and successful outcome. | The same work requires more tokens, more retries or more capable models than the validated baseline. |
| System Limits | Aggregate volume, rates, concurrency, queues, retries and external API use. | Correlated spikes, growing retry patterns, sustained near-capacity operation or behaviour consistent with loops. |
| Human Oversight | Alerts, approvals, escalations, response times and unresolved incidents. | Approvals become routine, alerts are ignored, operators cannot explain behaviour or response times deteriorate. |
| Auditability | Completeness of telemetry, decision records and intervention history. | Actions cannot be reconstructed, telemetry gaps grow or the system’s explanation is unsupported by independent evidence. |
Phase 4
Intervene before stopping
Detection is useful only when the system has a defined response. The appropriate intervention should be proportional to the risk, increasingly restrictive as confidence falls, and enforced outside the agent wherever possible.
| Condition | Response |
|---|---|
| Early or isolated deviation | Record the event, alert the owner and continue under closer observation. |
| Sustained drift | Reduce transaction rate, token budget, concurrency or model capability. |
| Sensitive or uncertain action | Require human approval before execution and preserve the full decision context. |
| Localized abnormality | Suspend the affected tool, workflow, agent or customer environment while the rest of the service continues. |
| Immediate or systemic risk | Revoke credentials, stop external actions and trigger the emergency kill switch. |
| Before restart | Diagnose the cause, update the operating envelope and repeat the relevant validation. |
Control-plane requirements
- The agent cannot override, suppress or silently modify its own controls.
- Credentials and tool access can be revoked independently of the agent runtime.
- Interventions can be scoped to a customer, workflow, capability or model rather than forcing a system-wide outage.
- Every restriction and stop event creates an auditable record.
- The agent has a safe escalation path through which it can pause and surrender authority when confidence is insufficient.
Phase 5
Recover, explain, and recalibrate
Stopping or restricting the system is not the end of the incident. Before service resumes, the team should understand what happened, what was affected, whether the implementation moved outside the operating envelope, or whether the original baseline was no longer valid.
| Review area | Questions to answer |
|---|---|
| Scope | Which customers, workflows, data, systems and external parties were affected? |
| Cause | Was the event caused by implementation failure, model behaviour, prompt or policy changes, tool failure, environmental change, legitimate growth or an outdated baseline? |
| Authority | Did the agent act within its defined permissions, and were those permissions appropriate? |
| Detection | Which signal first showed the problem, which signals were missed, and how early could the event reasonably have been detected? |
| Intervention | Was the response fast enough, proportionate enough and sufficiently targeted? |
| Evidence | Can the team reconstruct the sequence of actions and decisions independently of the agent’s explanation? |
| Restart | What has changed, which tests have been rerun, who has approved the restart and what additional monitoring will remain in place? |
Required feedback loop
Every incident should result in an explicit decision about whether to update the PRD, implementation controls, CI scenarios, production thresholds, operational ownership or customer communication. The operating envelope should be versioned alongside the product so that the definition of reasonable behaviour changes deliberately rather than drifting silently.
Operational record
Minimum governance record
For each AI product or materially distinct agent workflow, retain a concise record containing:
- The product purpose and accountable owner
- The authority granted to the agent and the systems it may access
- Expected transaction, token, cost and outcome ranges
- Warning, restriction and stop thresholds
- Required telemetry and evidence
- The intervention and escalation path
- The people authorized to stop and restart the system
- The date and result of the most recent validation
- The current operating-envelope version
The central principle
The kill switch matters, but it is not the management framework.
The real work is defining what reasonable behaviour looks like, making it measurable, validating it before deployment, detecting when it changes, reducing authority as confidence falls, and preserving the ability for a human-controlled system to stop the agent completely.