---
title: "Unchanged ChatGPT Ad Copy Is Not a Control Group"
description: "The same ChatGPT ad copy can render differently, reach different conversations, and produce misleading comparisons. Here’s what a credible manager should prove."
url: "https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a35e5725-8f82-4ebe-8a3e-837e832b2451.png"
published: "2026-11-10T06:09:48.647Z"
modified: "2026-11-10T06:09:48.699Z"
---

November 10, 2026 · 10 min read

# Unchanged ChatGPT Ad Copy Is Not a Control Group

[](https://groas.com/author/alexander-perelman)

[Alexander Perelman](https://groas.com/author/alexander-perelman)Personally Tested

Head Of Product @ groas

[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Cartoon of a proud ad manager guarding an untouched ad under a glass jar while his office floods and customers wander off, captioned "We didn't change a thing."](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a35e5725-8f82-4ebe-8a3e-837e832b2451.png)

In this article

1. [First, separate the submitted copy from the rendered ad](#first-separate-the-submitted-copy-from-the-rendered-ad)
2. [Next, separate delivery from buyer response](#next-separate-delivery-from-buyer-response)
3. [Then check whether the conversions are comparable](#then-check-whether-the-conversions-are-comparable)
4. [Limited reporting leaves several explanations open](#limited-reporting-leaves-several-explanations-open)
5. [A fair comparison is not yet a causal test](#a-fair-comparison-is-not-yet-a-causal-test)
6. [What I require from whoever manages the budget](#what-i-require-from-whoever-manages-the-budget)
7. [Apply the model to a win, not just a bad month](#apply-the-model-to-a-win-not-just-a-bad-month)

**Keeping your ChatGPT ad copy unchanged does not keep the advertising experience unchanged.** If your manager points to untouched copy as proof that their optimization caused a better month, they have skipped the part that needs proving.

Joe Martinez described the less cheerful version: a strong month on ChatGPT Ads, followed by a flat month with the same copy and the same signals on his end. Bids and budgets moved, but he had no clear view of what buyers actually saw. I treat that as one practitioner’s report from [Optmyzr’s PPC Town Hall](https://www.optmyzr.com/ppctownhall/how-should-ppc-teams-split-budget-beyond-goodle-ads/), not proof that every account behaves that way.

Its value is narrower. It exposes the gap between what an advertiser submits and what a buyer experiences. Rendering can change. Delivery can change. The time buyers have had to convert can change. Reporting may show the resulting number without revealing which mechanism moved it.

I want the mechanism before the victory lap. Here is how I work through it.

## First, separate the submitted copy from the rendered ad

Take a running example: say I am evaluating ChatGPT Ads for a B2B scheduling tool. The headline stays untouched for two months. That tells me what remained in the editor. It does not establish what buyers read.

In notes from a self-serve build, the character counter allowed 50 characters for titles and 100 for descriptions, while the panel warned that [copy may truncate above 24 characters in the title and 48 in the description](https://www.oligamy.marketing/blog/chatgpt-ads-how-to-advertise-on-chatgpt-2026). OpenAI’s guidance reproduced in those notes recommends titles of 16 to 24 characters and body copy of 32 to 48.

The distinction is simple: the box can accept text that the surface does not show in full. If the useful part of the scheduling-tool headline gets cut off, the buyer does not experience the complete promise sitting safely in the editor.

Those same notes describe [creative enhancements enabled by default unless advertisers opt out](https://www.oligamy.marketing/blog/chatgpt-ads-how-to-advertise-on-chatgpt-2026), allowing changes to headlines, descriptions, translations, images, and destination URLs without a per-version preview. That adds another gap between submitted material and delivered experience.

The Town Hall report describes a more visible warning: editing live ads surfaced a [yellow notice that the ads were eligible but would truncate well before the written count](https://www.optmyzr.com/ppctownhall/how-should-ppc-teams-split-budget-beyond-goodle-ads/). Eligible is not the same as readable. The editor is not the customer.

For my scheduling-tool example, I therefore start with two checks:

- What evidence do I have of the ad as it actually rendered?
- Were creative enhancements enabled, and did that setting change?

**Submitted copy is an input, not a record of exposure.** Before calling it unchanged, check the render and the enhancement setting. If the rendered version is unavailable, mark that gap as unknown rather than quietly assuming it away.

## Next, separate delivery from buyer response

Now suppose the scheduling-tool ad renders identically in February and March, but CPA doubles. That still leaves two different questions:

1. **Delivery:** Which conversations did the ad reach, and at what price?
2. **Response:** What share of the people reached clicked and converted?

Identical creative holds neither one fixed.

ChatGPT Ads delivery matches conversation context and intent rather than a keyword bought by the advertiser. The auction is described as a [relevance-weighted second-price auction using conversation context and intent alongside landing-page, creative, and context hints](https://www.searchenginejournal.com/6-months-into-chatgpt-ads-advertisers-still-dont-know-what-good-looks-like/582123/). Those hints are not keywords, and they do not guarantee delivery.

For the scheduling tool, February might contain more buying conversations and March more troubleshooting conversations. Longer threads might replace fresh questions. The same proof point might fit one set of conversations and miss another.

These are possible explanations, not things I can claim happened just because CPA rose. That distinction is the whole job.

Price adds another layer. A higher CPC does not, by itself, identify more competition, weaker relevance, or a change in available inventory. It records a more expensive click. The explanation still needs evidence.

**The same ad can buy a different mix of opportunities.** Before judging whether the message became less persuasive, separate changes in delivery cost from changes in response. Where reporting cannot explain the mix, leave the cause open.

## Then check whether the conversions are comparable

Even if rendering and delivery look stable, the comparison can fail after the click.

One [$608 self-serve test reported 72 ad clicks but only 16 AI Assistant sessions in analytics](https://www.indoormedia.com/blog/we-spent-608-testing-chatgpt-ads-heres-what-happened/). The team also recorded 12 thank-you page views it could not attribute, while phone calls were invisible without call tracking.

That does not tell me the missing clicks were worthless. It tells me the available measurements did not connect cleanly. An ad click, an analytics session, a thank-you page view, and a qualified opportunity are different observations. Calling all of them “results” does not make them interchangeable.

The scheduling tool adds a timing problem. February clicks might have six weeks to become pipeline while March clicks have six days. Comparing their current CPA treats unfinished buying decisions as finished ones.

I want three things aligned before taking that comparison seriously:

- **Cohort definition:** Compare clicks from equivalent-length periods, rather than mixing different groups.
- **Conversion definition:** Use the same outcome on both sides, ideally something downstream such as qualified pipeline rather than a chat click.
- **Maturity window:** Give both groups the same time to convert. If the comparison uses 30 days, both groups get 30 days.

This is not an elaborate testing framework. It is how I avoid comparing one month’s finished work with another month’s work in progress.

**Equal maturity makes a comparison fairer; it does not make it causal.** First make sure the numbers describe comparable outcomes. Then ask what those numbers can prove.

## Limited reporting leaves several explanations open

That last question matters because the platform may show the movement without showing its cause.

Six months into ChatGPT Ads, OpenAI [said it had no performance benchmarks across advertisers, industries, or campaign types](https://www.searchenginejournal.com/6-months-into-chatgpt-ads-advertisers-still-dont-know-what-good-looks-like/582123/). The reporting described there also lacked Auction Insights, impression share, and competitive reporting.

The [$608 self-serve test](https://www.indoormedia.com/blog/we-spent-608-testing-chatgpt-ads-heres-what-happened/) reported spend, impressions, clicks, CTR, and CPC or CPM, but no record of the actual prompts or conversations and no geographic delivery split. Those figures can show that traffic became more expensive. They cannot establish whether buying conversations became scarcer or competitors became more aggressive.

Return to the scheduling tool’s doubled CPA. At least four explanations remain possible:

- The rendered ad lost useful text and persuaded fewer buyers.
- Delivery shifted toward lower-intent troubleshooting conversations.
- Clicks became more expensive because of auction conditions.
- Recent clicks had not yet had time to become qualified pipeline.

A change log helps, but only within its reach. It records what the manager changed. It does not automatically record conversation mix, auction pressure, every rendered variation, or future conversions.

That is why “we changed nothing” is such a weak explanation. It describes the manager’s actions, not the whole system.

**When four causes fit one number, a credible report keeps four hypotheses open.** I would rather read an honest unresolved explanation than a confident story reverse-engineered from a chart.

## A fair comparison is not yet a causal test

A before-and-after comparison with untouched copy feels controlled because something stayed still. Unfortunately, it may not be the thing that explains the result.

For the scheduling tool, February buyers are not a holdout for March buyers. They arrived at different times, potentially through different conversations and auction conditions. Giving both groups 30 days to convert fixes the maturity mismatch. It does not remove the other differences.

[Incrementality compares treatment with a comparable holdout](https://www.aryma.ai/blogs/incrementality-testing), using the holdout to estimate baseline demand. That addresses a different question from platform ROAS: not merely what revenue was attributed to ads, but what happened because of them.

The question also determines the comparison. A holdout without advertising addresses the contribution of advertising. A properly controlled comparison of enhancements on versus off addresses the contribution of that setting. One does not automatically answer the other.

Possible designs include a geographic holdout, a paused audience slice, or an enhancements comparison with rendering checked on both sides. But I would not prescribe any of them just because the name sounds rigorous. The design has to be feasible, comparable, and capable of measuring the outcome. Holding one setting steady is not enough.

Thin volume and missing delivery information can make that difficult. The authors of the $608 test made the small-sample warning concrete: one $7,585 sale making the numbers look like 20 to 1 was [signal, not proof](https://www.indoormedia.com/blog/we-spent-608-testing-chatgpt-ads-heres-what-happened/).

I can still make a budget decision from imperfect evidence. I just cannot sell that decision as an established causal finding.

**Name the comparison before claiming the cause.** If the evidence is a before-and-after observation, call it that. It can be useful without pretending to be an experiment.

## What I require from whoever manages the budget

I want every important conclusion labeled with one of three words:

- **Observed:** What the available records show. For example, spend rose 18% while clicks stayed flat.
- **Inferred:** An explanation with its supporting reasoning and limits attached. For example, delivery may have shifted toward lower-intent conversations, but the available reporting cannot confirm that.
- **Unknown:** What the evidence cannot establish. Whether rivals bid up the same conversations, whether an unseen rendered variation changed response, or whether recent pipeline simply needs more time.

These are not three grades of competence. They are three different relationships to evidence. A provider who refuses to write “unknown” will eventually fill the space with something more expensive: certainty they have not earned.

![A ChatGPT-style answer showing the same ad headline in full in one panel and truncated in another](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c5866b1f-a806-4017-9aaa-1c858480a950.png)

Then I want a budget decision, not just an explanation of why explanations are difficult.

Say the account spends $20k a month, with $3k allocated to a ChatGPT test. Before that test starts, I want the manager to state:

1. How rendered ads will be checked, with weekly checks where the evidence is available.
2. Whether enhancements are enabled.
3. Which conversion counts, such as a sales-qualified lead (SQL), rather than a chat click.
4. How long each cohort gets to reach that outcome.
5. What would keep the test funded and what would trigger the kill rule.

The point is not to demand perfect visibility from a platform that does not provide it. The point is to stop missing visibility from becoming an excuse for loose decisions.

That is the management standard I expect from [groas](https://groas.com/): continuous checks, actions logged with reasoning, and a named strategist accountable for the outcome rather than a dashboard left for the client to interpret. Accountability means explaining both the decision and the limits of the evidence behind it.

**A manager does not need to explain every swing to earn the budget. They need to distinguish what they saw from what they think happened.** Ask for the render, the comparison, and the unknowns.

## Apply the model to a win, not just a bad month

Now take the mirror image of the opening story. Suppose the scheduling tool’s CPA falls 27% in three weeks, the copy stays untouched, and the agency claims its optimization worked.

I would work through the same layers in the same order:

1. **Rendering:** Did buyers see the same ad? Could enhancements have shortened a headline, or could truncation have changed? Without rendered evidence, that remains open.
2. **Delivery:** Did the account buy more buying-intent conversations and fewer troubleshooting conversations? That could lower CPA without making the message more persuasive. It is a hypothesis until the evidence supports it.
3. **Measurement:** Are the conversions defined identically? Did week-one clicks have 21 days to become SQLs while week-three clicks had four? If so, the cohorts are not ready for the comparison being claimed.
4. **Causality:** What comparable group separates the claimed optimization effect from everything else that moved?

Suppose the manager then compares **SQLs at 30 days of maturity**. That removes the timing mismatch. It does not settle rendering, delivery mix, or attribution of the improvement to the manager’s work.

I might keep the test slice alive while those questions remain unresolved. A promising observation can justify another decision without proving the explanation attached to it. The report should say exactly that: observed, inferred, unknown.

**An unchanged ad can teach me something. It cannot become a control group just because nobody edited it.** Any provider who does that work earns the test budget. Any provider who points at untouched copy and claims credit has told me exactly how they will explain the next flat month.

## Frequently Asked Questions

### Does keeping ChatGPT ad copy unchanged prove the ads stayed the same?

No. Unchanged copy only records what stayed in the editor. ChatGPT ad copy can render differently, truncate, or be modified by creative enhancements, so what buyers actually saw may change even when the submitted text does not.

### How many characters can ChatGPT Ads headlines and descriptions have before they get truncated?

The self-serve build's character counter allowed 50 characters for titles and 100 for descriptions, but notes from that build warn that copy may truncate above 24 characters in the title and 48 in the description. OpenAI's guidance recommends titles of 16 to 24 characters and body copy of 32 to 48.

### What are ChatGPT Ads creative enhancements and can they change my ads?

Creative enhancements are a ChatGPT Ads setting, enabled by default unless advertisers opt out, that allows changes to headlines, descriptions, translations, images, and destination URLs without a per-version preview. Check whether enhancements are enabled and whether that setting changed before judging results.

### If my ad copy is the same but CPA doubled, what could have caused it?

Identical creative does not hold delivery or buyer response fixed. ChatGPT Ads delivery matches conversation context and intent, so the same ad may reach a different mix of buying versus troubleshooting conversations, or clicks may simply have become more expensive. A higher CPC only records a more expensive click; the explanation still needs evidence.

### What should I check before comparing ChatGPT Ads results from two different months?

Check three things: compare clicks from equivalent-length periods, use the same conversion outcome on both sides such as qualified pipeline rather than a chat click, and give both groups the same maturity window, for example 30 days each. Equal maturity makes the comparison fairer but does not make it causal.

### What reporting limitations does ChatGPT Ads have for advertisers?

Six months in, OpenAI said it had no performance benchmarks across advertisers, industries, or campaign types, and the reporting lacked Auction Insights, impression share, and competitive reporting. A self-serve test also found no record of actual prompts or conversations and no geographic delivery split, so reported figures cannot establish why performance moved.

### Is an untouched-ad before-and-after comparison a valid control group?

No. Untouched copy does not make a before-and-after comparison a causal test, because earlier buyers are not a holdout for later buyers and conversation mix or auction conditions may differ. Incrementality compares treatment with a comparable holdout, using the holdout to estimate baseline demand, and the design must be feasible, comparable, and capable of measuring the outcome.

### What should I ask a manager running a ChatGPT Ads test to commit to before it starts?

Before the test starts, ask how rendered ads will be checked, whether enhancements are enabled, which conversion counts such as a sales-qualified lead rather than a chat click, how long each cohort gets to reach that outcome, and what would keep the test funded or trigger the kill rule. Conclusions should be labeled observed, inferred, or unknown.

## Run Your Marketing On groas

Plug ChatGPT, Claude or any AI agent into groas and let it do the work of a whole marketing team: $19 a month flat with every feature included, and autopilot whenever you’re ready.

[Start For $19](https://groas.com/pricing)

[Or have groas run it for you](https://groas.com/done-for-you)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","description":"groas is the marketing infrastructure for your own AI: it connects your AI to all your marketing and gives it proprietary marketing data collected 24/7. Plug Claude, ChatGPT or any AI agent into groas and run all your marketing from it, for a tiny fraction of the usual cost, with the groas app there whenever you want to see more. It is the layer ChatGPT, Claude and any AI assistant run a business’s online marketing on. Through groas an assistant can do anything you could do natively in Google Ads, Bing Ads, Meta Ads and ChatGPT ads, on your website and in AI search, and, where the owner switches it on, can run connected campaigns by itself, so AI agents do the work a whole marketing team used to. groas also gives those agents large-scale proprietary marketing data it collects around the clock, on any schedule the owner chooses: every ad competitors run on Google and Meta, their landing pages and full funnels with the offers and prices in them, Reddit customer research, AI search answers from ChatGPT, Google AI Overviews, Gemini, Claude and Perplexity, and SEO data from Semrush and Ahrefs, backlinks included ($19 a month, flat, with everything included, for unlimited teammates on one shared workspace and unlimited ad accounts and websites, delivered as an MCP server). Separately, its done-for-you service has senior people and proprietary quantitative systems run paid ads, SEO and AI search for businesses, and white label for agencies, from $5,000 a month.","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group#webpage","url":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group","name":"Unchanged ChatGPT Ad Copy Is Not a Control Group","description":"The same ChatGPT ad copy can render differently, reach different conversations, and produce misleading comparisons. Here’s what a credible manager should prove.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a35e5725-8f82-4ebe-8a3e-837e832b2451.png"},"breadcrumb":{"@id":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group#breadcrumb"},"dateModified":"2026-11-10T06:09:48.699Z","mainEntity":{"@id":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Unchanged ChatGPT Ad Copy Is Not a Control Group","item":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group#article","url":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group","mainEntityOfPage":{"@id":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Unchanged ChatGPT Ad Copy Is Not a Control Group","description":"The same ChatGPT ad copy can render differently, reach different conversations, and produce misleading comparisons. Here’s what a credible manager should prove.","datePublished":"2026-11-10T06:09:48.647Z","dateModified":"2026-11-10T06:09:48.699Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a35e5725-8f82-4ebe-8a3e-837e832b2451.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"copy, chatgpt, conversations, different, delivery, unchanged, comparison, unchanged chatgpt","wordCount":2050,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"My background includes Goldman Sachs and Stanford Computer Science.","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/an-unchanged-chatgpt-ad-is-not-a-control-group#faq","mainEntity":[{"@type":"Question","name":"Does keeping ChatGPT ad copy unchanged prove the ads stayed the same?","acceptedAnswer":{"@type":"Answer","text":"No. Unchanged copy only records what stayed in the editor. ChatGPT ad copy can render differently, truncate, or be modified by creative enhancements, so what buyers actually saw may change even when the submitted text does not."}},{"@type":"Question","name":"How many characters can ChatGPT Ads headlines and descriptions have before they get truncated?","acceptedAnswer":{"@type":"Answer","text":"The self-serve build's character counter allowed 50 characters for titles and 100 for descriptions, but notes from that build warn that copy may truncate above 24 characters in the title and 48 in the description. OpenAI's guidance recommends titles of 16 to 24 characters and body copy of 32 to 48."}},{"@type":"Question","name":"What are ChatGPT Ads creative enhancements and can they change my ads?","acceptedAnswer":{"@type":"Answer","text":"Creative enhancements are a ChatGPT Ads setting, enabled by default unless advertisers opt out, that allows changes to headlines, descriptions, translations, images, and destination URLs without a per-version preview. Check whether enhancements are enabled and whether that setting changed before judging results."}},{"@type":"Question","name":"If my ad copy is the same but CPA doubled, what could have caused it?","acceptedAnswer":{"@type":"Answer","text":"Identical creative does not hold delivery or buyer response fixed. ChatGPT Ads delivery matches conversation context and intent, so the same ad may reach a different mix of buying versus troubleshooting conversations, or clicks may simply have become more expensive. A higher CPC only records a more expensive click; the explanation still needs evidence."}},{"@type":"Question","name":"What should I check before comparing ChatGPT Ads results from two different months?","acceptedAnswer":{"@type":"Answer","text":"Check three things: compare clicks from equivalent-length periods, use the same conversion outcome on both sides such as qualified pipeline rather than a chat click, and give both groups the same maturity window, for example 30 days each. Equal maturity makes the comparison fairer but does not make it causal."}},{"@type":"Question","name":"What reporting limitations does ChatGPT Ads have for advertisers?","acceptedAnswer":{"@type":"Answer","text":"Six months in, OpenAI said it had no performance benchmarks across advertisers, industries, or campaign types, and the reporting lacked Auction Insights, impression share, and competitive reporting. A self-serve test also found no record of actual prompts or conversations and no geographic delivery split, so reported figures cannot establish why performance moved."}},{"@type":"Question","name":"Is an untouched-ad before-and-after comparison a valid control group?","acceptedAnswer":{"@type":"Answer","text":"No. Untouched copy does not make a before-and-after comparison a causal test, because earlier buyers are not a holdout for later buyers and conversation mix or auction conditions may differ. Incrementality compares treatment with a comparable holdout, using the holdout to estimate baseline demand, and the design must be feasible, comparable, and capable of measuring the outcome."}},{"@type":"Question","name":"What should I ask a manager running a ChatGPT Ads test to commit to before it starts?","acceptedAnswer":{"@type":"Answer","text":"Before the test starts, ask how rendered ads will be checked, whether enhancements are enabled, which conversion counts such as a sales-qualified lead rather than a chat click, how long each cohort gets to reach that outcome, and what would keep the test funded or trigger the kill rule. Conclusions should be labeled observed, inferred, or unknown."}}]}]}
```
