Impact Mapping an ML Bet

December 05, 2026 · 7 min read

Greenbox has 12,000 subscribers across Perth, Melbourne and Brisbane. Pause rate has moved from 2.9% in October to 3.4% in January, sixty more paused accounts a month than the run-rate would predict, a chunk of revenue softly bleeding off the side of the business. Sam flagged it in two ops reviews. Patricia Osei asked about it at the January board. It is on the wall, and Priya walks in on Tuesday with a plan to fix it.

The plan is clean, which is part of the problem.

“Two weeks on the data,” Priya tells Lee in the kitchen on Monday. “Three weeks of modelling. Two weeks of shadow mode. Live by the end of March. Kai’s already scoped the training run.”

Lee, washing a mug: “Who sees the output?”

“Sam’s team. They get a weekly list of at-risk subscribers. They call them.”

“And say what?”

Priya pauses mid-sentence. “Whatever they usually say.”

Lee puts the mug down. “Let’s do a Thursday. Bring Sam. Bring Marcus for the first hour. I’ll bring the pen. I want an afternoon before you cost the training run.”

Priya had not planned to spend an afternoon on this. She has had six weeks of we are doing things properly this time ringing in her ears since the last all-hands and she recognises the tone Lee is using. She goes to find Sam.

Thursday, 2 p.m.

The big meeting room. Sam, Priya, Kai, Tom, Jas, Charlotte. Anika on the laptop from Melbourne with the Yarra just visible through her window. Marcus for the first hour, with the specific expression of a commercial director who is about to be told that the thing he thought was a model is actually a business strategy.

Lee writes one line on the board and underlines it twice.

Why do we care that the pause rate is up?

Priya waits. She can tell Lee is setting something up and she does not want to be the first answer into the trap.

Marcus, half-interested, scrolling through his phone: “A paused subscriber becomes a cancelled subscriber about a third of the time within two months. If the pause rate stays elevated, the cancel rate follows with a lag.”

Lee: “That’s the mechanism. I’m asking about the people.”

Sam, quieter: “Some of the pauses aren’t life events. Some of them are us. A substitution the subscriber hated. A delivery that arrived in three pieces. A note that landed wrong.” She says the last one carefully. Maya isn’t in the room. Everyone still knows who she means.

Lee writes underneath his question:

Goal: catch preventable pauses before they happen.

He circles preventable in red, and the circle sits there for the rest of the afternoon doing quiet work.

Four actors, forty minutes

Lee does not draw the impact map. He asks the room to, which is slower and better.

Who can actually move this? turns out to be four people, and the room mostly agrees on them within ten minutes.

Sam’s team, obviously, the ones who would act on any signal the team produced. At-risk subscribers themselves, whose behaviour is what the bet is trying to shift. Farm operations, because if the signal cluster says this neighbourhood of subscribers all got the same bad swede week, the fix is upstream of Sam’s phone calls. And Marcus’s team, because somebody has to measure whether any of this worked, and Marcus is going to ask.

The behaviour changes take another twenty minutes and are less tidy. Sam can’t say what her team would actually say on a phone call. She says, and she is honest about it: “I don’t want them reading a script. But we don’t have a playbook and I’d rather not let them improvise a cold call to a grieving subscriber.” Priya nods, carefully.

Anika, on the laptop: “The subscriber behaviour we want is stays engaged, doesn’t pause, or pauses and comes back. Which one depends on the reason. Which we don’t know yet.”

Priya writes DON’T KNOW YET in block capitals on the corner of the board and nobody objects.

The farm-ops branch is the one Kai had not seen coming. Priya flags it: “If we see five subscribers in the same week all showing the same drift pattern, that’s not five phone calls, that’s a supply chain signal. It goes to farm ops, not Sam.” Kai writes it down in his own notebook. He doesn’t say anything for a while.

Marcus’s branch is shorter and sharper. “I need a control group,” he says. “A holdout, a random slice of the at-risk list we agree not to call, so there’s something to measure our outreach against. If you don’t give me one from day one I’ll believe whatever you tell me and we’ll both be wrong. Build the experiment before you build the thing.”

Lee writes holdout in red. He circles it.

What Priya came in wanting to ship

At the hour mark, when Marcus is putting his coat on, Lee turns to Kai.

“Three deliverables. You walked in with three. What were they?”

Kai does not need to check his notes. “Model. Dashboard. Slack integration that pings when a risk score crosses a threshold.”

“Which of the three,” Lee says, “makes any of the behaviour on that wall happen?”

Sam answers before Kai. “The list. Not the dashboard.”

Kai: “What’s wrong with the dashboard?”

“My team phones people. We don’t look at dashboards. We look at ticket queues and inboxes. If the list is in a dashboard, the list is for you, not for us.”

Kai sits with that for a second. “Right. Yeah. The dashboard was for me.”

Lee: “And the Slack integration?”

Kai: “Also for us. So we’d know the model was firing.”

“Does the model firing, in Slack, change anyone’s behaviour on the wall?”

“No.”

“Okay. They’re not deliverables, they’re reassurances. Put them in the bottom drawer. You can take them out again if they earn it.”

Kai writes bottom drawer on a Post-it and folds it into his shirt pocket. He will stick it to his laptop when he gets back to his desk. He does not take it off for the rest of the week.

What the wall had forgotten

By half past three the whiteboard has an actor map, a set of behaviour changes, and exactly one deliverable Kai still recognises. The room is quiet. Jas has not said much all afternoon.

She says it now. “We’re using preventable like it means something. It doesn’t yet. If we train a model on all pauses, it’ll learn to predict the ones we can’t do anything about. Life events, house moves, divorces. Those aren’t signal, those are noise, and if we chase them we’ll phone people on the worst day of their year.”

The word Ruth does not get said. It does not need to.

Sam nods slowly. “We need a definition before we have training data. I’d rather skip a week on the model than get this wrong.”

Priya, who started the afternoon wanting to cost a training run, hears herself say: “Next Thursday. Example mapping. What counts and what doesn’t.”

Charlotte, near the door: “And the call script. Sam, do you have a senior on your team you trust to write the first draft with you?”

“Dina. She’ll hate it but she’ll do it.”

“Book the time.”

The shape Priya walked out with

Priya took a photograph of the whiteboard on her phone before she left the room. It had five deliverables on it in the order they would now be done.

Define preventable pause, with examples people in the room had argued about. Write the outreach playbook with Dina so the phone calls didn’t invent themselves. Design the experiment, with a holdout Marcus’s team would sign off. Wire up a feedback path so every call fed something back, into farm ops if it was supply, into the definition if it was a new pattern, into the model once the model existed. Then, and only then, build the model.

Kai said, as they walked down the corridor: “This is week six of a project we scoped as three weeks.”

Priya: “Yeah.”

“The model’s going to be the easy bit.”

“Yeah. That’s kind of the point, isn’t it.”

Kai nodded. He had the bottom-drawer Post-it in his shirt pocket still. He had not quite got round to sticking it on his laptop yet.

Priya did not drive straight home. She pulled off at Point Resolution and sat in her car with the windows down. The river was the colour a river is in Perth in early February, which is to say too bright to look at directly. She was there for about twenty minutes. She did not think about anything in particular. A container ship she had watched approach for half of those minutes finally crossed the line of the pylon she had been using as a marker. At five past seven she started the car and drove home.

What counts as a preventable pause, and where the model could quietly hurt someone.

These posts are LLM-aided. Backbone, original writing, and structure by Craig. Research and editing by Craig + LLM. Proof-reading by Craig.