Hypothesis -> Test -> Analyze -> Iterate?

You've probably sent a campaign, looked at the open rate, shrugged, and moved on. That's not optimization. That's just sending. The Hypothesis-Test-Analyze-Iterate cycle is what turns each send into something you actually learn from.

Here's how the four stages work, with a real example woven through.

Hypothesis

A good hypothesis isn't "let's try a shorter subject line." It's a specific, testable prediction with a reason behind it. Something like: "Removing the deadline date from our subject line and replacing it with a countdown phrase will increase opens by 8%, because urgency framing feels more personal and less promotional."

That format matters. You've named what you're changing, what you expect to happen, and why you think it'll happen. If you can't explain the why, your hypothesis isn't ready yet.

Test

Now you design the experiment. Split your list into two groups of equal size. Group A gets the control (your current subject line). Group B gets the variation (the countdown version). Everything else stays the same: send time, from name, body copy, CTA.

One change per test. That's the rule. If you change the subject line AND the preheader AND the send time, you won't know which variable moved the needle. Also, make sure your sample is large enough to mean something. A 200-person list isn't going to give you statistically reliable data. Most ESPs (like Mailchimp or Klaviyo) need at least 1,000 recipients per variant before the results are worth trusting.

Run the test to completion before you declare a winner. Checking results after two hours and calling it done is how you get fooled by noise.

Analyze

The hypothesis said opens would go up by 8%. Maybe they went up by 3%. Does that count as a win? Maybe. But before you decide, ask three questions.

• Is the difference statistically significant, or could it be random variation? (Most A/B testing tools will tell you this directly.)
• Did any segment behave differently? Maybe mobile readers responded strongly but desktop readers didn't move at all.
• What else changed? Click rate, conversion rate, unsubscribes? A subject line that lifts opens but tanks clicks isn't actually a win.

Write down what happened, including the results you didn't expect. Those surprises are often the most useful data points.

Iterate

Here's where the cycle compounds. If the countdown phrasing worked, your next hypothesis isn't "let's do that again." It's "now let's find out whether personalized countdown language (using the subscriber's first name) outperforms the generic version." You're building on what you know.

If the countdown phrasing didn't work, that's equally useful. Now you hypothesize why. Was the audience already familiar with your urgency tactics and tuned them out? Test a different angle entirely, like specificity over urgency ("3 items left in your size" vs. "Last chance").

The real value of this cycle isn't any single test result. It's that each round leaves you with a clearer picture of your audience than the round before. After 10 cycles, you're not guessing anymore. You're working from a body of evidence specific to your list, your tone, and your readers.

If you're not sure where to start testing or what to prioritize first, the next question in this series covers how to prioritize what to test. And if you want to make sure your results are actually being captured and shared with your team, there's a question on documenting test learnings that's worth reading alongside this one.