A while ago, we released A User Story, an explanation of our Aampe’s algorithms, told in the form of a storybook. We wrote A User Story because we wanted anyone, from any background, to be able to understand our systems well enough to comprehend why they work.
Now we’re releasing “How do I know it works?”. This is a much more technical version of A User Story. It’s more detailed than a white paper but, we hope, much more accessible than an academic paper. The purpose of this document is to delve into the weeds of how our systems learn.
The concept of lift lies at the core of our systems. Lift is a measure of return on investment, and there’s no such thing as ROI without attribution - this paper explains why and in what cases we feel confident attributing user behavior to the notifications we manage. We show not only how we calculate lift in a way that doesn’t rely on naïve comparison groups, but also show how we can use that lift metric to assess the different parts of our system to understand what works well, and what still has room for improvement. More than that, we explain why our system can be trusted to automate decisions of when, to whom, and about what to send notifications.
