Introducing Counterfactual Models: A Deeper Look at Message Timing

Each user gets an AI Agent

Our agents fully personalize message timing - each user gets his or her own programmatic agent, and that agent decides what day of week, what time of day, and how frequently to message (plus lots of other stuff).

Why not send at a general best time?

It's not uncommon to look at aggregate results and note that one or two days and times tend to perform better than the others - Friday at 9:00pm gets only a 0.11% purchase rate, but Sunday at 6:00pm gets a 0.21% rate. "Why," our customers sometimes ask, "don't the agents just send all of the messages on Sunday at 6:00pm?"

That's a fair question. The table below is the answer. The observed purchase rate is what the rate *was*, but that doesn't tell us what the rate *could have been* if the agents had made different choices.

‍

So we built a model to estimate those counterfactuals.

I looked at when agents actually messaged users, and then joined in a bunch of the weights that our agents used to make the timing decision, then bumped all of that data up against the weights we had at that time for all of the *other* timing options the agents could have chosen. The resulting counterfactual model estimates what would have happened if we'd sent all messages at just one time rather than personalizing.

The columns in the "actual" section of the table shows the actual number of messages and responses (purchase), along with the rate (responses / messages). The "counterfactual" section shows model estimates under the assumption that we send everything at just one time. The "comparison section" shows the difference in counts between the summed responses of the actual section and the responses for each row of the counterfactual section, and the percent difference in the two sections' rate columns.

The results

While there were two cases where our actual rates were a little lower than the counterfactual rates, in every case, the total number of expected responses was between 322k and 1.8 million purchases less than what we actually got by letting the agents personalize timing. If we average the difference, personalization does 20% better - equating to 1.1 million more purchases in this case - than just picking the "best" time.

There is no general "best" time.

Each individual user has a best time, and some users happen to overlap in their preferences. If you're picking a single or even a handful of "bests", you're missing out on tons of value.