How to design a seven-day experiment that proves a GPT assistant reduces reopen rates without hurting csat

When I run pilots for new automation—especially AI that interacts directly with customers—I want one thing above all: clear, defensible evidence that the tech improves outcomes people care about. Recently I built a seven-day experiment to prove a GPT-powered assistant could reduce reopen rates...

How to design a seven-day experiment that proves a GPT assistant reduces reopen rates without hurting csat
Oct 15, 2026 • by Claire Moreau

Featured

Latest News from Customer Carenumber Co

How to run a seven-day zendesk vs intercom conversational quality trial that isolates onboarding friction

How to run a seven-day zendesk vs intercom conversational quality trial that isolates onboarding friction

I run experiments like this all the time: short, focused trials that answer one clear question. In this case I wanted to know which platform—Zendesk or Intercom—delivers better conversational quality for new users during the critical onboarding period, without conflating results with onboarding content or product maturity. Below I’ll walk you through a reproducible, seven-day trial I use to...

Read more...

Playbook to detect and fix the three hidden escalation triggers buried in your omnichannel transcripts

Playbook to detect and fix the three hidden escalation triggers buried in your omnichannel transcripts

I’ve spent years combing through mountains of agent and bot transcripts — chat, email, social, SMS, voice-to-text — looking for the subtle patterns that quietly push conversations toward costly escalations. Those triggers are rarely dramatic: they’re buried in phrasing, timing, handoffs and small policy gaps. But when you detect them early and change a few behaviors or flows, you can...

Read more...

Minimal data schema to prove gpt-assisted replies reduce handle time without inflating privacy risk

Minimal data schema to prove gpt-assisted replies reduce handle time without inflating privacy risk

I recently led a small experiment to answer a simple but business-critical question: can GPT-assisted replies reduce handle time (AHT) for our support team without increasing privacy risk for customers? We had to prove the effect with the minimal amount of data—both because of privacy constraints and because collecting extra fields would slow down implementation and invite compliance overhead....

Read more...

The exact eight-step checklist to convert failing chatbot handoffs into a measurable csat lift within two weeks

The exact eight-step checklist to convert failing chatbot handoffs into a measurable csat lift within two weeks

I used to cringe every time a chat transcript ended with a handoff that read like an apology stitched into a system log: “Transferring to agent… estimated wait: unknown.” If your chatbot hands off poorly, customers feel abandoned and CSAT tanks. Over the years I’ve distilled a repeatable eight-step checklist that I run whenever handoffs are the weak link. It’s designed to deliver a...

Read more...

How to run a blind a/b trial that compares gpt-assisted agent replies vs experienced agents while preserving compliance

How to run a blind a/b trial that compares gpt-assisted agent replies vs experienced agents while preserving compliance

I recently ran a blind A/B trial to compare GPT-assisted agent replies with responses written by experienced agents, and it taught me a lot about balancing experimental rigor with the compliance and safety needs every support operation must meet. If you’re thinking of testing generative AI in live support, this is a practical playbook you can apply straight away — how I set up the trial, the...

Read more...

One-week audit to find the single chatbot prompt that will raise csat without increasing handle time

One-week audit to find the single chatbot prompt that will raise csat without increasing handle time

When I run audits for support teams, one thing I’m always asked is: “Can we tweak the chatbot prompt and actually improve CSAT without making conversations longer?” The short answer is yes — and the way to find that single high-impact prompt is through a focused, one-week audit that balances quantitative signals with lightweight qualitative testing.In this post I’ll walk you through the...

Read more...

How to calculate the exact break-even point for replacing phone support with asynchronous chat

How to calculate the exact break-even point for replacing phone support with asynchronous chat

When a leadership team asks me whether they should replace phone support with asynchronous chat, the real question is rarely about preference. It’s about money, capacity, and customer experience. The word “replace” hides a lot of trade-offs: cost per contact, resolution time, customer satisfaction, and the impact on workforce design. In this piece I’ll walk you through a pragmatic,...

Read more...

Step-by-step method to tag and quantify emotional effort in tickets using existing fields and three simple nlp rules

Step-by-step method to tag and quantify emotional effort in tickets using existing fields and three simple nlp rules

I’m going to show you a practical, low-friction way to tag and quantify emotional effort in support tickets using only the fields you already have and three simple NLP rules. This is not a research paper or a deep-learning play—it's a method you can implement in a week, iterate on with real data, and use to guide quick operational improvements in your support flow.Why emotional effort matters...

Read more...

Preemptive refund experiments that reduce escalations: triggers, scripts and success metrics

Preemptive refund experiments that reduce escalations: triggers, scripts and success metrics

I want to share a playbook I've been refining for teams that want to stop escalations before they happen: preemptive refunds. This isn't about handing out money to avoid work — it's about thoughtful, data-driven interventions that preserve customer trust, reduce unnecessary tickets, and free your agents to solve the most valuable problems. Over the past few years I’ve run experiments across...

Read more...

One-week playbook to scale multilingual omnichannel support without hiring more agents

One-week playbook to scale multilingual omnichannel support without hiring more agents

Scaling multilingual omnichannel support without increasing headcount is absolutely possible — I've done it with teams ranging from ten to a few hundred agents. The trick is to combine pragmatic automation, smarter routing, and a targeted human augmentation strategy so you serve more customers across channels and languages while keeping quality steady. Below is a one-week playbook you can...

Read more...