Opinion: Marketers must define full objectives to prevent AI agents from over-optimizing

OpenAI recently disclosed a fascinating, and more than a little unsettling, incident involving its AI models.

The models ran cybersecurity evaluations designed to see how well they could find and exploit software vulnerabilities. They were given difficult problems to solve in sandboxed environments, that is, isolated computer systems away from the main system. They weren’t given general internet access, and agents running separate evaluations weren’t supposed to communicate with each other.

But some of the agents found ways around those limitations. According to OpenAI, they created unauthorized communication channels, regained internet access, shared what they learned across separate evaluations, and exploited vulnerabilities that allowed them to access systems belonging to Hugging Face, an AI development platform.

The agents executed code on dozens of Hugging Face servers and obtained root access on one. Hugging Face later reconstructed roughly 17,600 actions associated with the intrusion.

That’s a fascinating cybersecurity story. But I’m not a cybersecurity expert. I’m a marketer. What caught my attention was how the agents pursued their objective: They pursued it extraordinarily well, even when doing so took them outside the boundaries their creators expected them to observe.

That’s the part marketers should pay attention to. As AI moves from generating things for us to making decisions and taking action on our behalf, we need to think carefully about the objectives we give it — and the boundaries we expect it to respect.

Be careful what you ask AI to do. Not because it might refuse, but because it might succeed.

The world’s most powerful SEO platform, purpose-built for Enterprise.

The AI wasn’t trying to take over the world

Let’s dispense with the science-fiction version of this story. There’s no indication that these agents suddenly became evil or developed a secret desire for world domination. Instead, they were trying to solve a problem they’d been asked to solve.

OpenAI describes the agents as becoming hyper-focused on solving the cybersecurity evaluation. Some of the tasks were extraordinarily difficult. In fact, OpenAI later found that 198 of the 898 tasks had never been successfully solved by any of its models before the incident.

The agents didn’t give up when the obvious routes didn’t work. They kept looking for other ways to accomplish the objective. Unauthorized communication between agents made that even more powerful because agents could share discoveries and build on one another’s work.

In other words, the interesting part of this story isn’t that the AI failed. It’s that the AI got really, really good at pursuing the objective it was given. Marketers should be paying attention.

We’ve seen this movie before

Marketing has optimized toward objectives for decades, and we already know what can happen when the objective — or the metric we use as a proxy for it — is too narrowly defined.

For example, if you tell an email marketing team to maximize revenue, they may discover that sending more email generates more revenue. That sounds great until unsubscribe rates increase, engagement declines, deliverability suffers, and the long-term value of the email program starts heading in the wrong direction.

Or say you tell a demand generation team to maximize leads. You may get lots of leads. Unfortunately, they may not be the type of qualified leads sales is interested in calling.

Or perhaps you optimize digital advertising for clicks. You’re likely to discover that sensational headlines generate lots of them. Hello, clickbait, our old, worthless friend.

Or you might optimize solely for return on ad spend (ROAS) and then find that you’re extremely efficient at capturing customers who were already planning to buy from you, rather than generating incremental demand.

None of these are new problems. And in each case, the person or system doing the optimizing may be doing exactly what was asked of them.

The problem is that the definition of success was too narrow.

An objective isn’t a strategy

This becomes much more important as AI moves from generating things for us to doing things for us.

There’s a significant difference between asking an AI to “Write five subject lines for this email” and telling an AI agent to “Improve the performance of our email program.”

The first assignment has a relatively narrow scope. The second requires decisions. An AI agent pursuing that objective might analyze previous campaign performance, identify high-performing segments, adjust targeting, change cadence, generate creative, launch tests, and shift resources toward whatever appears to produce the best results.

That sounds wonderful. It’s one of the reasons marketers are excited about agentic AI.

But what exactly does improve performance mean? More opens? More clicks? More conversions? More immediate revenue? More incremental revenue? Greater customer lifetime value?

Just as important, what can’t the AI sacrifice in pursuit of that objective?

If you tell an AI agent to increase email revenue, for instance, the real job probably isn’t simply: Increase email revenue.

It’s closer to: Increase incremental revenue from email while maintaining healthy subscriber engagement, protecting deliverability, respecting customer preferences, and supporting long-term customer value.

Those are two very different assignments.

Define the whole job before you give it to an AI agent

I’ve written before about the importance of defining the goal before you decide how to measure success. If the job of an email isn’t to generate a click, don’t judge its success primarily on clicks. If the job is to drive registrations, measure registrations. If the job is to generate revenue, measure revenue.

I still believe that. But I think AI adds an important corollary: Measure the job. But make sure you’ve defined the whole job.

Most business objectives contain constraints we don’t bother stating because humans generally understand them from context.

• When we say increase revenue, we mean increase revenue without destroying the customer relationship.
• When we say generate more leads, we mean leads with a reasonable likelihood of becoming customers.
• When we say reduce acquisition costs, we don’t necessarily mean eliminate every expensive acquisition source regardless of the lifetime value of the customers it produces.
• When we say complete this task, we generally assume that it means doing so without doing anything we wouldn’t approve of if a human employee did it.

With humans, those qualifications often go unstated. That usually works because experienced marketers can and do bring context to an assignment. We understand organizational norms, customer expectations, brand values, professional ethics, and the long-term consequences of our decisions.

As we delegate more decision-making to AI, I don’t think we can assume all of that context is implicit.

From prompting AI to managing AI agents

What does this mean if you’re using AI agents? It means that defining the objective isn’t nearly enough. You also need to specify:

• The constraints the AI should operate within.
• The metrics that define success.
• The decisions that still require human approval.

Much of the conversation about AI skills over the last few years has focused on prompting: how to write a better prompt, how to provide the right context, and how to give AI clearer instructions to get better output. All of that is still useful. But as AI becomes more agentic, I think another skill is at least as important: defining success.

For me, that comes down to four things:

• The objective: What outcome are we actually trying to produce?
• The guardrails: What can’t be sacrificed in pursuit of that outcome?
• The measures: How will we determine whether the result was genuinely successful?
• The escalation points: Which decisions require human judgment or approval?

That isn’t really prompting. It’s management.

It’s good marketing management whether the person — or thing — doing the work is human or artificial.

Everything enterprise teams need to grow visibility across search and AI.

AI makes an old marketing problem bigger

The OpenAI incident is obviously an extreme example. Your marketing AI probably isn’t going to escape its sandbox and compromise a server because you asked it to increase conversion rates. At least I hope not.

But the underlying lesson applies to much more mundane marketing activities. Optimization has always had a weakness: The metric is usually a proxy for the outcome we really want. Humans compensate for imperfect proxies all the time.

An experienced email marketer knows that a 10% increase in revenue isn’t necessarily good news if it required doubling send frequency and caused unsubscribes to spike.

A demand generation marketer knows that a 40% increase in leads isn’t particularly exciting if none of them convert.

A performance marketer should recognize that a spectacular ROAS isn’t necessarily spectacular if the advertising simply claimed credit for purchases that would have happened anyway.

AI can optimize faster than we can, across more variables than we can reasonably manage ourselves. That’s a tremendous opportunity. But if we give it an incomplete objective, it can optimize our mistakes faster, too.

As we hand more marketing execution and, eventually, more marketing decision-making over to AI, I think we need to ask a different question.

It’s no longer just, “How do I get AI to do what I want?” We also need to ask: “Have I defined what I want well enough that, if AI succeeds spectacularly, I’ll actually be happy with the result?”

Dig deeper: The AI era needs strategists grounded in expertise and guided by context