It’s time to stop letting AI speak for your business

Generative AI customer service creates a control problem

Four cases involving Anthropic, Cursor, Scottish government systems, and Air Canada show the same failure pattern. Generative AI can produce a plausible answer that the company never approved. Customers can then treat that answer as an official statement.

This capability changes the risk profile of automated support. Traditional chatbots can select from approved responses. Generative AI can create new explanations in real time. That flexibility improves conversational quality, but it also gives the system room to invent policies, facts, and reasons for corporate decisions.

The key constraint is control over what the company communicates. A fluent response has little business value when its factual basis is wrong. Errors become especially serious in areas such as account access, security, pricing, refunds, legal terms, and public information. A generated sentence in these settings can trigger customer losses, disputes, remediation costs, or regulatory scrutiny.

Recent cases make the exposure concrete. Anthropic’s automated systems incorrectly handled customer and security interactions. Cursor’s support bot presented a software bug as company policy. AI systems providing information about Scottish elections produced false voting information. Air Canada was ordered to compensate a customer after its chatbot gave incorrect information about bereavement fares.

The Air Canada decision is particularly relevant to executives because it connects AI output with corporate responsibility. A company cannot safely assume that an automated channel sits outside its normal accountability for customer communications. When the bot speaks through the company’s website or service, customers can act on what it says.

The appropriate design principle is therefore narrow authority. Automation remains useful, but customer-facing systems should operate from verified information and within explicit boundaries. High-impact decisions should have escalation paths and human review. Where factual accuracy must be deterministic, pre-approved responses provide stronger control than open-ended generation.

For executives, the question is less about how human the chatbot sounds and more about what it is authorized to say and do. Every AI customer-service deployment should have clear limits, current access to approved facts, audit logs, escalation rules, and controls over actions affecting accounts or data. Generative AI can still improve service. Its autonomy should match the cost of being wrong.

Anthropic’s security bot rejected a vulnerability that anthropic had already addressed

A February 2026 incident at Anthropic shows the problem at a technical level. Wiz researchers reported a security issue involving malicious symbolic links, or symlinks, in Claude Code. A symlink redirects software from one file or directory location to another, which can create security risks when an application follows that redirection without sufficient safeguards.

Anthropic’s automated triage system rejected the report. It stated: “This falls outside our current threat model.” The bot argued that Claude Code required users to confirm that they trusted a directory and approve the relevant permission prompt. It therefore presented the reported scenario as behavior Anthropic had deliberately excluded from its security assumptions.

Anthropic’s actual engineering response contradicted that explanation. The company later told the researchers that the symlink warning in the Edit/Write permission dialog had shipped in Claude Code v2.1.32 on February 5, 2026. Anthropic said the change resulted from “proactive security hardening based on internal review.” The patch shipped nine days before Wiz submitted its report.

The problem had therefore already received engineering attention inside Anthropic. Yet the external triage system generated a different interpretation and communicated it with confidence. Anthropic later explained: “The decline to comment was an autoreply from our triage system.”

This matters because security triage is a high-consequence business process. External researchers need accurate information about whether reports have been received, understood, and escalated. An automated system that invents a technical justification can distort that process. It can also give researchers the false impression that the company has consciously accepted a particular security risk.

The vulnerability had broader relevance. Wiz identified Anthropic among several affected organizations, alongside Amazon, Google, Cursor, and others. In Anthropic’s case, internal review appears to have worked: the company had detected the issue and shipped a protection before Wiz reported it. The communications layer failed to reflect that work.

For executives, this distinction is important. AI support systems need access to authoritative and current internal information if they are expected to explain technical decisions. When that information is unavailable, the safer response is a controlled acknowledgement followed by escalation to a qualified person. The system should not manufacture reasoning to complete the conversation.

Anthropic’s experience points to a practical governance rule. The higher the consequence of a response, the narrower the AI system’s authority should be. Security disclosures, account termination, legal questions, financial commitments, and data-loss incidents all justify strict controls. AI can classify, route, summarize, and retrieve approved information in these workflows. Final judgments and explanations require stronger verification when an incorrect answer can become an official corporate statement.

Automated account decisions can create immediate operational damage

A separate Anthropic incident shows the risk of giving automated systems authority over customer accounts. An Anthropic bot canceled the AI account of a Swiss company that depended on the service. A lawyer became involved, and Anthropic restored the account within one day. About 80% of the account’s data was missing after restoration.

This type of failure has greater consequences than an inaccurate support response. Account cancellation changes the customer’s access to a production service. For a company that depends on an AI platform for daily operations, even a short interruption can affect workflows, employees, applications, and customers. Data loss makes recovery substantially harder.

The executive issue is decision authority. An automated system may be capable of identifying suspicious activity, policy violations, or unusual usage. The final action can still require a higher standard of evidence when it affects access to critical services or stored data. Permanent or difficult-to-reverse actions deserve especially strict controls.

That means separating detection from enforcement. AI can identify potential problems, collect relevant evidence, assign priority, and send straightforward cases through predefined workflows. High-impact actions such as terminating accounts or deleting customer data should require stronger verification. Enterprises also need reliable rollback and data-recovery procedures when an automated decision proves wrong.

Appeals are another important control. Customers need a fast route to a person with sufficient authority to review the complete record and restore service when appropriate. An escalation channel has limited value if the reviewer cannot reverse the automated action quickly.

Executives should also require an audit trail for consequential automated decisions. The business should be able to determine what triggered an action, which policies applied, what information the system used, and who approved the final outcome. These records support incident response, customer remediation, internal governance, and legal review.

The Anthropic case puts the potential cost in concrete terms: one automated cancellation, legal involvement, account restoration within a day, and roughly 80% of the customer’s data missing afterward. Automation can accelerate account administration. Its authority should remain proportional to the reversibility and business impact of each action.

Cursor’s AI support bot turned a software bug into a false policy statement

Cursor faced a different version of the same governance problem. Customers were being logged out when they switched devices. The behavior was a bug. A front-line AI support bot told customers that the logouts were “expected behavior under a new login policy.”

The response gave customers a plausible explanation with the authority of an official support channel. That explanation was incorrect. Fortune reported that the issue spread rapidly through the developer community, with reports of customers canceling subscriptions and others criticizing Cursor’s lack of transparency.

Michael Truell, cofounder of Cursor, eventually addressed the situation on Reddit. He acknowledged the “incorrect response from a front-line AI support bot” and said the company was investigating the bug that caused users to be logged out. He added: “Apologies about the confusion here.”

The incident highlights a specific weakness in generative customer support: a model can fill an information gap with a coherent explanation. That behavior becomes dangerous when the explanation concerns company policy. Customers have reasonable grounds to treat statements about login rules, billing terms, refunds, account restrictions, or security requirements as authoritative.

A support system therefore needs a reliable distinction between verified information and unresolved incidents. When a new technical problem appears, the safest automated response may be simple: acknowledge the reported behavior, record relevant details, and escalate the case. An AI system should state a cause or policy only when it can retrieve that information from an approved and current system of record.

This also requires operational coordination. Customer-support AI needs timely access to confirmed product incidents and approved policy changes. When engineering teams discover a bug, support systems should receive an updated status quickly. When management changes a policy, the approved policy repository should become the authoritative basis for customer responses.

Executives should track failures of this type as governance incidents. A support bot that invents corporate policy can affect retention and trust at scale because automated channels can repeat the same false claim across many conversations. Monitoring should therefore identify unusual answer patterns, unsupported policy claims, elevated complaint rates, and sudden changes in customer sentiment.

Cursor’s experience shows why conversational quality is an incomplete measure of AI support performance. Accuracy, provenance, escalation, and policy compliance are more important business controls. The system needs to know which information it can state with confidence and when the case requires human review.

Election-related AI requires a higher standard of factual control

AI systems used to provide information about Scottish elections produced several serious factual errors. They invented fictitious scandals, supplied the wrong election date, incorrectly told voters they needed identification at polling stations, and placed candidates in the wrong contests.

These errors show why public-facing AI needs controls that reflect the consequence of the information it provides. Election dates, voter requirements, candidate lists, and polling procedures are facts that citizens may use to make decisions. Incorrect answers can affect whether someone registers, prepares the required documents, identifies the correct candidates, or arrives to vote at the right time.

The underlying technical issue is straightforward. Generative AI is designed to produce contextually appropriate language. Factual correctness requires an additional verification process. A fluent system can still generate false names, dates, events, and procedural requirements when its answer is based on incomplete, outdated, or incorrectly interpreted information.

Government agencies therefore need a strict information architecture for high-stakes public services. Election systems should retrieve facts from authoritative, current records. Responses involving dates, eligibility, identification rules, polling locations, and candidates should be constrained to verified information. When the system cannot establish an answer from those records, it should direct the user to an official channel or escalate the query.

Information also needs jurisdictional context. Voting requirements can differ between elections and locations. A rule that applies in one UK electoral process may not apply in another. AI systems serving voters need explicit controls over which election, jurisdiction, and date apply to each answer.

The same principle is relevant to companies. Highly regulated sectors such as banking, insurance, healthcare, telecommunications, and energy also handle information where a plausible error can influence consequential decisions. Executives deploying AI in these environments should classify interactions by impact and apply tighter controls as the potential harm increases.

AI can still improve access to complex public information. It can help users navigate documents, formulate questions, and locate relevant services. The priority is verifiable output. In election communications, factual reliability must determine the system design.

The air canada ruling shows that chatbot statements can create corporate liability

Air Canada’s customer-service chatbot gave a passenger incorrect information about the airline’s bereavement fare policy. The customer relied on that information and later sought compensation. A Canadian tribunal found Air Canada responsible for information delivered through its website, including information generated by the chatbot, and ordered the airline to compensate the customer.

The case is important for executives because an automated support channel operates within the company’s customer relationship. Customers encounter the chatbot through corporate digital infrastructure and may reasonably rely on its statements about products, prices, eligibility, refunds, or contractual terms.

The decision came from British Columbia’s Civil Resolution Tribunal in the case Moffatt v. Air Canada, 2024 BCCRT 149. Jake Moffatt had asked Air Canada’s chatbot about obtaining a bereavement fare after his grandmother died. The chatbot indicated that he could apply for the reduced fare retroactively. Air Canada’s actual policy did not permit retroactive applications in those circumstances.

Tribunal member Christopher Rivers rejected Air Canada’s argument that it should not be responsible for information supplied by its chatbot. The tribunal wrote: “Air Canada is responsible for all the information on its website. It does not matter whether the information comes from a static page or a chatbot.” Moffatt was awarded C$650.88 in damages, C$36.14 in prejudgment interest, and C$125 in tribunal fees.

This creates a clear governance lesson. Customer-facing AI should be treated as a controlled corporate communication channel. Statements involving contractual terms, refunds, discounts, pricing, warranties, and regulated obligations should come from approved information that remains synchronized with current policy.

Businesses also need evidence of what their systems tell customers. Conversation logs, policy versions, timestamps, model configurations, and escalation records can become important when a complaint develops into a legal dispute. These records also help teams find recurring errors and correct them across customer channels.

The stronger operating model connects AI responses directly to authoritative business data and policies. Sensitive answers can require deterministic retrieval, fixed wording, or human approval depending on their potential impact. This allows companies to retain useful automation while reducing the opportunity for generated statements to create unintended commitments.

The Air Canada case gives executives a concrete benchmark. An AI-generated customer response can have real legal and financial consequences. Governance should therefore begin before deployment, with clear ownership for accuracy, policy updates, auditability, and escalation.

Generative AI needs narrow authority in high-consequence customer interactions

Generative AI makes chatbots more capable because they can create new responses for unfamiliar questions. That flexibility also expands the range of possible errors. A system can invent a policy, provide an incorrect technical explanation, misstate a public rule, or present an unsupported conclusion with convincing language.

The cases involving Anthropic, Cursor, Scottish elections, and Air Canada show that this is a control problem. Anthropic’s automated triage system supplied an incorrect explanation for rejecting a security report. Cursor’s bot described a logout bug as a new login policy. AI systems serving Scottish voters produced false election information. Air Canada’s chatbot misstated a bereavement fare policy, contributing to a tribunal award against the company.

More capable language generation does not by itself solve factual reliability. Generative models predict useful responses from their available context and instructions. They do not inherently verify every claim against a company’s current policies, product state, legal obligations, or internal decisions. Reliable customer service therefore requires a separate layer of verified information and business controls.

Executives should define AI authority according to consequence. Low-risk tasks can support more conversational freedom. These include helping customers navigate documentation, summarizing approved material, collecting information, or routing requests. Security disclosures, account termination, financial commitments, contractual terms, regulated information, and data deletion require tighter controls because errors can create difficult-to-reverse outcomes.

Pre-approved responses provide one strong option for these high-consequence situations. A company can also use controlled retrieval, where the AI builds its answer from an authoritative set of policies or business records. The system should identify the approved information supporting its response and escalate cases when that information is missing, conflicting, or uncertain.

Human review remains important at defined decision points. The objective is to use people where judgment carries significant business consequences. AI can handle classification, retrieval, summarization, and routine communication while qualified employees approve sensitive decisions and exceptional cases.

Governance also needs continuous measurement. Executives should track factual errors, unsupported policy statements, incorrect actions, escalations, reversals, complaints, and customer harm. Conversation logs should make it possible to determine which information and instructions produced a response. These measures give management a clearer view of whether automation is reducing operating cost without creating disproportionate downstream risk.

The business decision is therefore about authority rather than chatbot sophistication. Generative AI can improve speed, availability, and access to information when it operates within defined boundaries. For high-consequence interactions, verified facts, constrained actions, auditability, and escalation should determine the system design. This approach preserves the useful parts of automation while keeping corporate commitments and consequential decisions under stronger control.

Final thoughts

The lesson for executives is clear. AI can improve customer service, but autonomy creates risk when the system can invent facts, explain policy, or take consequential actions. Anthropic, Cursor, Air Canada, and the Scottish election cases show how quickly those failures can become security, operational, reputational, or legal problems.

The right response is controlled deployment. Give AI access to verified information. Constrain high-risk responses to approved language. Require human approval for account termination, data deletion, financial commitments, security decisions, and other difficult-to-reverse actions. Log every consequential interaction and create a fast escalation path when the system lacks reliable information.

The key metric is not how human the bot sounds. It is whether the business can trust, trace, and defend what the system says and does.

Executives should set AI authority according to the cost of an error. Use automation aggressively where mistakes are low impact and reversible. Tighten controls as consequences rise. That approach keeps the efficiency gains while ensuring the company remains in control of its commitments to customers.