Why email AI can meet targets and still harm business

Email AI can hit its target and still harm the business

An email AI model can predict correctly and improve its target metric while damaging the business. A model that identifies subscribers likely to leave, for example, still needs a sound rule for deciding how to contact them. For marketing and technology leaders, disappointing results therefore raise two questions: is the system failing to learn, or successfully optimizing a rule that harms the business?

“AI scales what you give it,” said Shar VanBoskirk of Forrester Research during her opening keynote on AI in email marketing at the ANA Email Excellence Center Committee Meeting in August. Forrester sells research and advice to businesses making technology decisions, so it has a commercial interest in demand for AI guidance. The familiar framing, “Garbage in, garbage out,” describes an input-quality problem recognized since the early days of machine learning. Its counterpart, “Misalignment in, misalignment out,” describes what happens when an objective leaves out important business conditions.

Automation magnifies both problems because faster execution leaves less time to notice missing variables, while fewer people involved means fewer opportunities for review and correction. AI agents, systems that can take actions automatically, are becoming more common. Their spread has renewed attention to how quickly significant unintended consequences can develop. Teams therefore need to evaluate prediction accuracy, improvement in the chosen metric, and business benefit separately.

Accurate predictions need sound rules for action

The gap between prediction and business benefit became clear when a brand deployed an AI model to reduce email subscriber churn, meaning subscribers leaving its email program. Once the model was running, the brand discovered that it was suppressing campaigns to new subscribers. The prediction was correct: those subscribers were the group most likely to churn. But the model used that prediction to withhold contact from them.

The harmful decision came from applying a shared rule to new and established subscribers, even though they needed different treatment. Improving the prediction would have left that suppression rule intact, so the brand paused the model and later relaunched it with separate rules for the two groups. The correction changed what happened after a subscriber received a high-risk prediction. Accuracy became useful only when the business chose an appropriate response.

Choosing that response becomes harder when the target metric captures only the beginning of a customer’s experience. Subject-line models judged by open rates almost always favor vague, mysterious language, which can persuade someone to open an email while telling them little about its contents. Recipients may then feel misled or disappointed, causing clicks and conversions, the actions the business wants recipients to complete, to fall while unsubscribes and spam complaints rise. The open rate improves even as the email program’s broader results deteriorate.

To account for what happens after an open, teams can optimize subject lines using click rates, connecting the subject line’s promise with the email’s body content. Where conversion volume is sufficient, conversion rates extend that connection to the landing page and the action the business wants the recipient to complete. Recipients experience these steps in sequence, so the objective needs to account for how well each step fulfills the expectations created by the previous one.

Even a conversion-based objective can omit commercial value, as an unnamed national bank found in its credit-card operations. Its model reduced promotion of platinum cards because their conversion rates were comparatively low, favoring entry-level cards with higher uptake. Platinum cardholders, however, had substantially greater lifetime value, the value of the customer relationship over time, than entry-level cardholders. The bank changed the qualification rules governing platinum-card promotion to increase applications and issuance.

The bank’s adjustment shows why acceptance probability must be considered alongside the value of an accepted offer. Retail personalization adds another condition: a recommendation can fit a customer’s preferences while doing little to help the retailer sell stock in its stores and warehouses. Product recommendations therefore need to combine customer interest with existing inventory. A likely purchase can still be a poor commercial choice if the recommendation ignores what the retailer needs to sell.

Inventory is one condition; profitability is another, as grocery recommendations illustrate. A grocer can recommend popular national brands alongside cheaper store brands that carry higher margins. Customers’ responses over time should then determine whether later recommendations increase or reduce store-brand exposure. The business can thus adapt to individual behavior while accounting for the profit generated by the products it promotes.

Useful optimization requires connected data and informative campaigns

Those business conditions can influence a decision only if the system has usable evidence about them. Turning customer records into audience groups, personalized messages, and automated actions begins with resolving identities: determining which records and activities belong to the same person. Next comes collecting enough behavioral evidence for the intended use case. With those foundations in place, the business can draw insights to guide its decisions.

Even when the company holds the necessary information, the system making the decision may be unable to use it. Customer records can sit in separate applications, while weak integrations restrict their movement into email platforms or other systems that carry out campaigns. The model then operates without information the brand already has. This problem also affects loyalty programs and digital channels more broadly, making connections between systems part of the optimization work.

Forrester’s report The Underutilized Superpower Hidden in Loyalty Data shows how these barriers affect loyalty program professionals. In that group, 57% reported that poor integration prevented effective use of loyalty data, while 48% identified operational silos, teams or functions working separately, as an obstacle. These findings make a practical case for inspecting how loyalty, email, and other channel systems exchange information. They also point to organizational barriers that technical connections alone may leave in place.

Connecting systems can make existing information available, but missing behavioral evidence requires a different response because the relevant observation has yet to be collected. A team can wait for further observations. Or it can use three active methods, depending on whether it needs customers’ stated preferences, evidence from their actions, or additional information held elsewhere.

The first method, progressive profiling, collects information directly from customers through polls, quizzes, surveys, and preference centers. Because people explicitly provide these answers, they are zero-party data. A business can ask about a preference instead of waiting to infer it from behavior, adding information that can support the intended use case.

When a business needs behavioral evidence, experimentation produces first-party data through the audience’s reactions to different content, campaigns, and designs. The brand varies what people receive, observes their responses, and uses those responses to infer preferences. Data enrichment takes the third route by obtaining additional information about prospects or customers from external providers, described as third-party data. The choice depends on which collection process can fill the specific information gap.

Of these methods, experimentation most often causes brands difficulty, and send-time optimization shows why. Suppose a brand sends its daily campaign to a subscriber at the same time every day. The model can observe responses at that time, but it has no observations showing how the subscriber responds at alternative times. It therefore lacks the comparisons needed to determine whether another time would work better.

Continuing that schedule adds observations without necessarily adding the missing comparison, so the brand should vary send times deliberately and thoughtfully for a period. Responses to those alternatives give the model evidence to compare. Here, campaign policy limits learning, which means changing that policy is part of making optimization possible. A readiness review therefore needs to ask whether relevant information can reach the system and whether campaign practices generate the evidence the model needs.

Customer intent must be detected in time and matched with a proportionate response

Evidence collected over time still leaves a timing problem: the customer may be doing something different now. Understanding current context begins with observing present customer and prospect activity, then estimating what they are likely to do next. That assessment of intent must continue as activity changes. Earlier preferences and behavior can inform it, but current interactions determine whether an intervention remains appropriate.

Keeping the assessment current requires cross-channel systems that can observe activity and respond in real time. Batch systems, which process accumulated information at intervals, delay the business’s response to customer behavior. By the time the system acts, the intent that made the action useful may have shifted. For decisions that depend on current context, response timing is therefore an operating requirement.

Meeting that requirement is a near-universal difficulty because an expression of intent can disappear before the business has a chance to intervene. Teams unable to observe and respond promptly across channels face a practical limit on the opportunities they can act on. Historical insight can still help with planning, even when it arrives too late to guide a particular interaction.

Even when the signal arrives in time, the system must judge its strength before choosing a response. It first needs to distinguish meaningful activity from noise, then decide how much action the activity warrants. Scoring signal strength and matching the response to it makes that judgment explicit. Without this calibration, a weak indication of interest can trigger a response that exceeds what the customer’s behavior warrants.

The appropriate response can also change as the customer moves through the journey, because intent can strengthen or fade. A response that fits at one point may become excessive or insufficient after new behavior appears. The system must therefore reassess both the signal and its planned action throughout the interaction. Fast execution helps only while the action still fits the customer’s current behavior.

A clean-input test can help locate the failure

These requirements create several possible causes of poor results, so diagnosis should come before a decision to rebuild the model or its data infrastructure. The Mirror Test offers a focused check: give the model an ideal customer signal showing strong intent and inspect whether the output is useful and relevant. Then compare that result with the model’s behavior on production data, the information it receives in everyday operation. The comparison helps direct the investigation toward the model or its inputs.

An unidentified speaker gave the instruction: “Give your AI a perfect, high-intent customer signal and see what comes back,” then explained the interpretation: “If the output is generic or irrelevant, the model needs work. But if the model produces something sharp and useful on clean data, and then falls apart on real production data, the problem is the data.” Running the comparison before making larger changes gives the team a clearer basis for deciding where to work.

But the test has a limit: useful output can still serve a poorly chosen objective. As the subscriber-suppression, subject-line, and bank examples show, a good prediction or a higher target metric can lead to an unsuitable business action. A successful clean-input test therefore leaves a separate question: do the rules that turn the output into action account for the business conditions that matter to that decision?

Main highlights

• Align email AI with business value: Marketing owners need to evaluate prediction accuracy, target metrics, and business results separately. Rules governing AI actions should account for customer stage, lifetime value, inventory, and margins where relevant.
• Build the evidence optimization needs: Data and platform teams need to connect customer records across systems, while campaign owners design experiments that produce useful comparisons. Repeatedly sending emails at the same time gives send-time models little evidence about better alternatives.
• Match responses to current intent: Cross-channel systems need timely customer signals and rules that match each response to the signal’s strength. Reassess planned actions as customer behavior changes to keep interventions appropriate.
• Diagnose failures before rebuilding: Model owners can use the Mirror Test to compare output from an ideal, high-intent customer signal with output from production data. That comparison helps locate model or input problems; a separate review of decision rules establishes whether useful outputs lead to sound business actions.