Advertising Attribution: How to Know Which Cannabis Ads Are Driving Dollars

{}Introduction and Guest Background

Jake Litke: I feel like this takes longer than it used to. Okay, there we go. We should be live everywhere now. Hi everyone. Welcome to Cannabis Marketing Live. I am your host, Jake Litke, and today we have Alexander Morisse, who I've worked with over the years and is actually a pretty good close personal friend. So I'm very pleased to have him here and to have his brain to teach us some things. We are going to be talking about attribution today. We'll start off talking about some general high-level marketing principles with attribution, and then towards the end we'll get pretty nerdy, and we'll let Alex bring out his formulas and data science for us, and I'll try to keep up.

Alexander Morisse: I didn't bring my chalkboard.

Jake Litke: So what are we going to do? That's good. I'm glad you didn't bring your chalkboard or your notebook with all of your—

Alexander Morisse: You know, I've got the notebook. I can show that.

Jake Litke: So, okay, fair enough. Alex is a data scientist, he has a PhD in string theory, is that correct?

Alexander Morisse: That's right.

Jake Litke: Yes. We'll try to avoid string theory because that's kind of outside of the scope, but maybe you could talk a little bit about your path through data science, what you're doing now, and then we can get into it.

Alex's Journey From Academia to Data Science

Alexander Morisse: Excellent. Yeah, well first of all, just a pleasure to be here, and great to be with the audience that we're live streaming to, so welcome everybody, we're in for a fun ride here. My journey in data science was really — let me tell you an apocryphal tale — I spent many years in academia, I spent about 12 years, I did some really fun research, worked with some of the sharpest minds in the industry, and never thought I was going to do anything different. And then, you know, life sort of throws you — it sort of trips you up sometimes and you have to adapt. And a good friend of mine who was running analytics at a small company called Facebook said, hey, you know math, you should get into data science. So I asked him, how do I do that? And he said, you have to learn this language called Python. Of course, I'd never heard of any other programming language other than Fortran, Basic, and Mathematica. So this was back in 2010, I was using this thing called Google to figure out how do I start learning Python, actually bought a book on it and started coding in the terminal, in the Python terminal, so that was sort of my splash into it. Spent about six months transitioning from academia into learning Python, and then John got me an interview at Quid, and Quid was doing some really interesting, basically large-scale graph analytics over public data, and this was an incredible place for me to jump in and cut my teeth on how do I take in messy data, how do we turn that into signals, and then how do we turn that into actual products that can drive intelligence for our end clients.

Jake Litke: Yeah, we'll get into signal and noise probably quite a bit today. So you're out of the academia world for a while now, and you're fully commercialized as a data professional, and what are you working on right now, like, when you're not talking to me on the internet?

Graphistry and the Rise of Louie.ai

Alexander Morisse: Yeah, right now — exciting, it's been an exciting few years — started as head of AI at Graphistry, and Graphistry is a graph intelligence platform, it's also an open-source repo, you can go to pygraphistry on GitHub, and it's basically a very easy way to take a data frame, a messy data frame, throw it into the Graphistry API, and then get beautiful graphs out. You can get explicit graphs, like source-destination, make hypergraphs, so you can see all types of attributes. So imagine you have a very messy, or even potentially structured, data set that you're pulling back from Databricks or whatever your database is, now you can throw that into the graph, and you can basically visualize the schema of that database. This is super useful for behavioral data, user behavioral data, because it naturally clusters that into not only semantic clusters but potentially explicit clusters. So if you wanted to see users by gender, age, spend, things like that, very clear signal — but also, if you have more interesting tags on them, or actual vectors, you can then cluster on those vectors, and that happens all in the API, so it's a very powerful way to work on things. Now, the last few years we were developing some really heavy-duty technology around graphical neural nets, so basically taking networks that we're used to from PyTorch or TensorFlow, and then turning that over into the graph domain. That was going great, and of course what happened — right, we had the LLM revolution — so I told Leo, I said, hey, we should probably pivot on this roadmap and start developing sort of an LLM-based version of the knowledge engine that Graphistry is built on. So that pivoted into what's called Louie, you guys can check out Louie.ai, and Louie.ai is really a beautiful product, from the perspective of being able to talk to your databases, being able to do heavy lifting and analysis just by natural language, and then also being able to rerun workflows. So now anybody can come in and pretty much do almost any type of hard data science application workflow on sort of mesoscale data — not like the terabyte-size stuff, but definitely like million-row type of.

Understanding Data Frames and Large-Scale Ad Data

Jake Litke: So try to translate some of that a little bit — so first of all, we'll try to translate phrases that we're used to throwing around, but one thing that's important, you mentioned the word "data frame," right, so when you're dealing with large data, and when you're running advertising campaigns you're generally running millions of impressions, right, so you've got millions of points of signal across a bunch of geography, a bunch of publishers, different creative sizes — a data frame is essentially a spreadsheet, right, so you think of it as you've got an x and y axis, you generally have columns that are your dimensions and rows that are going to be your events, right, so each row is one advertising signal, or a purchase, and then your dimensions are going horizontally. And when you're dealing with large data, you can't do this stuff in Excel, it'll just break, right, so if you need to load 10 million rows, and look at all — you're also limited in Excel to basically, like, what I consider first-order analysis, right, very simple pivots and things like that.

Alexander Morisse: Yeah. Yeah.

Jake Litke: So there is a — when you, let's just take, let's say you've got 10 million ad impressions you've served over the last 90 days, okay, so it's a huge amount of data, there's no way you can look at it and get the insights that you want, so you have to use software to do that. Generally people are going to use some version of Python, they're going to put it into a large data store, it could be Databricks, it could be Snowflake if you want to spend a lot of money, it could be ClickHouse, or it could just be on your computer, right, you could have a CSV file with all these rows and then you can process it. Now, why do we want to look at all of this data? Well, first off — and I'm going to keep this around advertising, because that's what we focus on — if you're advertising, you're spending money, right, so every single one of those points of signal, you've spent money on that, and you want to take that data and understand how you want to spend your money better in the future, right. So let's take — we'll take a couple simple dimensions that are important, the most obvious one is publisher, right, so you're running ads, which publisher are you running them on, depending on what your brand is there, you get very different results from the type of content that your advertising is going to be on.

Alexander Morisse: Yeah. And I'll take another simple example — people play games on their phones, there's lots of opportunity to advertise there, people read the news, right, now the same person doing those two different things is effectively a different conversation, and they're in a different mindset, right, so another example I like to give is, if you're — let's say you're a business professional and you're a parent, right, when you are at the airport on a business trip versus when you're at the park with your child, your mind is in a completely different place, right, you care about different things, and you will engage with different content, right. So what you're doing and when you're doing it is important.

Why Context Matters in Advertising

Jake Litke: And you can change the outcome of your advertising, you know, 2, 3x, in terms of dollars in versus dollars out, if you look at this data and understand what's happening. Right, so that's the advertising, that's your engagement, you're sending a message out into the world. Now the part that we focus on a lot at this point is purchases — people make online purchases, right, with attribution models, we have ours, other people have theirs, Google has their own, Facebook has their own. What you're effectively doing is you're looking at the signal that you're seeing off of the engagement, and you're looking at the signal that you're seeing coming off of the purchase, right, so if someone orders something online, you get two signals there, and the way you do attribution is you measure how similar they are. And again, a very simple version would be, you can do it with cookies, although cookies are less important these days, you can do it with the actual event itself, so when you serve an ad, and you have an attribution tag on it, you get an actual electronic signal to your servers, and it has information in it, like IP address, device type, other things, and then you can use that to measure back.

Alexander Morisse: Yeah, go ahead.

Jake Litke: Yeah, go ahead.

Alexander Morisse: I mean, essentially, what I think the point that you're trying to drive across, if I could sort of add a little bit, is that many times these experiences are very context-dependent. Right, and depending on whatever context you're in, you can actually take what I would consider a very multimodal distribution, right, so relaxing at home watching something on TV, versus being at an airport thinking in terms of a business trip, or seeing some sort of billboard ad as you're driving into San Francisco, they're all sort of very different contexts. So what gets very interesting, in my mind, is finding the linear combination of that that really amplifies the experience, and many times, in the attribution modeling, it's very difficult to know exactly what affected what, and we know that there's a lot of synergy between the two — so for instance, CTV or DOOH can drive awareness at scale, and it lends itself to building trust and cognitive awareness and emotional readiness before the user ever searches or clicks. So those types of things are very interesting from a contextual perspective, and that's where we start thinking about applying the mathematics of Bayesian analysis, and these types of things, to those problems, so you can actually derive real signal from noise there.

Introduction to Attribution Modeling

Jake Litke: Yeah, and to set the table a little bit — a lot of people that haven't spent their career in marketing, you know, they start a business, could be a retail location, could be online, and most people think about advertising like Instagram ads, right, you run an ad for some widget and then you click on it and buy it, but that's not really how marketing works in the broader scope of things. You generally need to have a bunch of touch points with the consumer over a period of time, depending on what the purchase is. Right, so there's a lot of science around this, sometimes people say it's 10 times, sometimes people say it's 20 times, where someone has to see your brand that many times before it registers in their brain as a real thing, right. So when we're doing, let's say we're doing online DTC sales, right, someone's — what we see generally is a window of about 7 to 10 days between the first impression that you send someone and making a purchase, right. And the other thing that happens — again, a lot of people think, you know, I'm going to serve an ad, someone's going to click on it, they're going to buy something — most of the time what happens is you serve people ads as they're going about their life, and then they come to some sort of purchasing decision point, and then they go and make that purchase. And a lot of that traffic comes in through organic, right, they saw the name enough times, they go and search for it, and then they go and make the purchase. And this is where a lot of people get tripped up, because they over-attribute their organic traffic, because they don't understand the signal that happened leading up to that, right, someone searched for your name and they're like a new customer to you, how did they find out about your name to search for it?

Alexander Morisse: Right, if they're actually looking for your brand.

Jake Litke: That's right. So now you've got, let's say you've got 10 signal points in a 10-day period, and you need to decide, as a marketer, how much value you want to give to those signals, right, generally the closer the signal is to the purchase, is a stronger signal, right. But let's talk about some of the math on that, you mentioned Bayesian, which is probably a phrase that people have heard more than some of the other jargon that you might throw around, but let's just dive into what is Bayesian and why is it useful.

Bayesian Reasoning Explained

Alexander Morisse: Yeah, so the Bayesian — can be described, let me try it in a very simple lay person — think about it, Bayesian reasoning is what humans do very well, or at least most of the time. And what the mathematics is, is it's not necessarily about figuring out what exactly happened, it's figuring out the math around how you change your beliefs. Okay, I'll give you an example — you're at the office, your neighbor calls you and says, "hey, your house alarm is going off," right, what's the first hypothesis, thought, that you go through in your mind? You think, "oh, my house is getting broken into." So now you're freaking out, you jump in the car, you drive over the bridge back home, and you turn on the radio just to clear your mind, and now you hear there was a 4.3 earthquake in the region, right. Now what happens, what happens to the prior belief that you had, right? So something in your mind, right, you can imagine you have a bunch of thoughts or hypotheses about what the state of the world really is, but those can modulate and change given new information that comes in. So Bayesian mathematics is the mathematics around that type of reasoning. And of course it goes very deep, it touches string theory, it touches all these very deep things, but essentially that same exact math that I was doing as a grad student, postdoc, professorship, was exactly the same math that then became incredibly valuable when we were building all the things we built.

Applying Bayesian Math to Attribution

Alexander Morisse: So from that perspective, if I can write down a calculus or mathematics that describes how I can update my belief of the state of the world, then that's a very valuable system for me to understand things like sequential, or attribution modeling, through a very noisy data set, right. So let's say I know that I spend x amount of dollars on advertisement out in the wild, right, so brand awareness — now that brand awareness I know is producing some sort of signal in the wild, and then I've got some other system out here, which is, let's say, Google search, right, so now I see that there's ad spend that happens, let's say it's a million dollars, just for a round number, and then there's some lag, and then all of a sudden I start noticing that there's some upward trend in people searching my name, right, there's a clear correlation between those things, and so you can actually now write down the mathematics of those things, and you can understand, for every dollar spent, what's the change in the slope on what happens there, because we know that that's going to turn into money or revenue downstream.

Jake Litke: Yeah, that makes sense. One of the examples I like to give, which resonates well with people when you talk about this, is sometimes you see an ad for pizza and then you want pizza, right, you're like, "oh, that looks good, I want that," however, you might not be in a position to have pizza at that moment, but you're like, "you know what, I'm going to get pizza on Friday," right, so it's tying those two signals together, and then weighting it.

The Pizza Example: Tying Signals Together

Jake Litke: Right, so let's say you have a purchase and it's $100, okay, almost always that's going to happen because multiple events occurred, there's potentially awareness advertising that occurred, they may have then searched for your product, or the brand itself, like "I want pizza" versus "I want Pizza Hut," they're going to get search results back, or they're going to get ChatGPT results back, which we're seeing a lot more traffic coming from that, and then you have multiple placements that happen in that consumer experience, right, if you're doing your SEO right, you'll show up organically, but generally you're also doing paid search, right, so that you buy that top spot so that your name comes up. Having both of those positions is reinforcing to the consumer that you are a serious company, right. And there's another conversation where people are like, "oh, I don't want to buy my own name because I feel like that's wasted money, because if someone was searching for me, then they would just hit the organic result." However, if you type in any major brand name, like just type Nike into Google, Nike is going to show up right at the top, sponsored, right, they're buying that position because that individual impression actually has a nominal cost. But the ability to show up both in organic and paid search, there's a psychology behind that for consumers, and they'll click on one of the two, right, they might click on the sponsored one, maybe, "I don't want to give Google money," if they actually know what's happening in the background, so they click on the non-sponsored one.

Alexander Morisse: Yeah.

Jake Litke: But either way, you've taken those two positions, right. So that means that you've got the $100 purchase, you spent some money on advertising somewhere — could be CTV, could be digital, could be a billboard — then you spend a little bit of money on the paid search result, and then you've also spent money somewhere to get the organic result to the top also, right. So you now have three different things you're paying for, and you have a $100 purchase.

Balancing Ad Spend Across Channels

Jake Litke: Now we get into a pretty highly subjective and controversial topic, which is, if you have $100, and let's just say your margin is 50%, right, so now you have $50 of profit, you want to make sure you're spending less than $50 for that, right, collectively across those channels. But how do you decide — is the advertising worth 50%, is the paid search worth 20% — and I'm just making numbers up, it's just a conversation to think about, and you have to account for that, you have to account for all of those things. If you're going to be running a real — if you're trying to build a real brand and do real marketing, you need to hit people across multiple channels, and then you need to basically take your best guess as to how you want to allocate the investment across those channels. So, let's use my hypothetical situation with those relatively simple data points, how would you guide someone in that, like, what do you look at, and how do you try to get some semblance of the truth?

Alexander Morisse: Yeah. Yeah, within this — the sphere of algorithmic analysis, right, you take historical data and you try to infer, basically, what interactions statistically contribute to the conversion. So one thing that I know MediaJel can do, because they have those channels, is we can basically take all those different signals, put that within a statistical model, and then basically do the moral equivalent of regression over that model, so you can do some sort of feature importance, and figure out that, in general, I have 20% in this channel, with an error bar of 5%. Many times — and this is where it gets difficult, and we found this many times — is that basically the standard deviation is much greater than the mean, right, and then you don't actually know.

Jake Litke: All right, so you're going to have to — you've got a wide range of audience here, so standard deviation, how much something varies, right, mean, what's the average, what do you expect, right. So now if I say to you, what's the temperature in San Francisco today, right, and you're like, "well, normally it's about 20°C, right, okay, well, 60 degrees Fahrenheit, sorry."

Alexander Morisse: Yeah, yeah, I'm in physics brain, so Celsius was always the one. But yeah, so — what if it's now 95, right, so that is a very huge leap from what you'd expect.

Confounding Variables and Statistical Noise

Alexander Morisse: Okay, so the same thing, in these types of attribution modeling, is that you have essentially a huge set of confounding variables, and confounding variables means variables that you might collect but you don't actually know if they're producing the right signal that you understand or care about. And many times, when you do this type of analysis, you might find that the variable in question fluctuates so much compared to what you would expect, that it actually doesn't become a useful variable to explain the distribution, or explain the data. So, in general, just to answer your question, in general, there is a bit of a nuanced art here, right, and there are ways that you can test this using different methods, to figure out exactly what is a good mix, but in general, just start 20/20/20, something like that, or 40/20/40, and then you can sort of adjust from there.

Jake Litke: Yeah, I'll give you a pretty topical example, 4/20 just happened, right, so that creates a — I mean, when — I totally missed that, it's April 20th, oh yeah, I'm going to miss it as a holiday, I guess.

The 4/20 Attribution Challenge

Jake Litke: Oh yeah, so, obviously we have a lot of advertisers that are running 4/20 campaigns, so the issue with 4/20 is that, if you look statistically across all of the cannabis companies that we work with, you're going to have potentially triple, quadruple, like, a lot — like the sales, you look at the chart, the sales is just going to have a massive spike in it, right, and this is something that we are having conversations about now, which is, okay, well, we ran advertising, but also you've got this anomaly on this day or this weekend where the sales are going to quadruple on their own, essentially, if you don't do anything. But you still want to advertise, because you want people to come to your store, and everyone else is advertising, and the sales volume in that day, in the category, is much much higher, so you want to try to capture that revenue. So you're like, "okay, well, I spent some tens of thousands of dollars running marketing and my sales quadrupled, but what really happened, right, how much?" And that's an example where you have a lot of noise in the signal, and that's ultimately the biggest problem to solve with attribution, is eliminating noise from your signal, and you have both false positives and false negatives, right.

Alexander Morisse: Yeah, and I mean, this is also from the perspective of just pure attribution, right, where you're thinking, "was it the last click that I cared about," but also, from the perspective of, there's another thing here which is just brand awareness, right, and sometimes that's just worth spending, because that can actually lift the tide of the entire process, it may not be something that you see directly that day, because there's already such a great uptick, you're just happy making whatever the money that you made, but essentially, that day is a special day to get your brand out there, irrespective of anything.

Brand Awareness as Billboards: The Airport Analogy

Jake Litke: Yeah, that's — you know, it's funny, I was traveling with my family, and I was talking to my daughters, and we were at the airport — we'll just talk about the airport more, which is a fascinating place — and here's one thing that's interesting, we were sitting there, and they're like, "why is there a Prada and Louis Vuitton store in the airport, right, there can't be that many people buying purses."

Alexander Morisse: Yeah, I mean, there is, but those stores are expensive to be there, right, airport real estate is expensive, those are effectively billboards, right, those billboards, primarily for brand awareness, they're not really there to sell bags, and it's an expensive place to be, right. And I think if you kind of look at the world, like airports are good billboards, anywhere where you've got high traffic, and you actually take a look around and see what's happening within your physical world, where advertising is, there's a reason that it's there, right.

Jake Litke: And it's not — and this is getting more into the same thing I was saying before, which is, a lot of people come into marketing thinking "I'm going to serve an ad and someone's going to click on it and buy something," but that is, like I said earlier, almost never the scenario that happens, it takes time, and it takes lots of touches, that's why we call it multi-touch attribution, across many channels. And we always encourage advertisers we work with — because we do a few different things, but we're focused entirely on digital, right, so if it has a screen on it we can serve ads on it — but that doesn't mean that billboards are not effective, it doesn't mean that — you know, it's actually really effective, and people forget about, is direct mail, right, we don't send direct mail, but we work with some companies that do, and we're able to measure the return, and direct mail for a dispensary usually has like a 5x ROAS, you send out 10,000 mailers and you spend, you know, six, seven thousand, whatever your vendor is charging.

Direct Mail Attribution Case Study

Alexander Morisse: So I'm curious, how do you actually measure it?

Jake Litke: We integrate the POS data, right, so you actually really have a hard signal on mailers, because you know the physical addresses that you sent them to, and then if someone comes in and makes a purchase — another thing that's unique, at least about dispensaries, is you generally have the addresses of people, because they have to show their ID, right, now that's personal information, you can't necessarily use that to market, but you can use it to know that I sent out — and this is a real world example, we just did this in Ohio, I think — sent out 10,000 mailers, the mailing company did, and then you can see over the next, call it 30 days, if you want to use a 30-day attribution window, how many people at which address purchased, right, and then you can plot that on a map, which becomes interesting, because then you can look at, well, these particular zip codes within my geographic area over-indexed, right, this — they were statistically different, they were above — you know, it could be a one or two standard deviation — why? Why did that happen, right? And a common, simple example is the income in the zip code, right, and your marketing message — and this can come down to very specific things, right, so, like, when you send a marketing message that is "get 30% off your purchase," or you send a marketing message that says "get $20 off," or "get a buy one get one free," those two messages resonate differently with different income levels, right, and with people's average order value — if you're someone that spends $20 at a time, 30% is not a very big number of $20, right, so in that case, getting a free product, which could have the same net value, right, the consumer has a very different engagement level, right, and it can come down — this, I'm using some simple examples, right, income and discounts, there's a correlation there. So run the campaign, then you realize that, okay, well, I'm going to send out two different mailers next time, I'm going to send one with this kind of discount to these zip codes, and a different one to these zip codes, and making those little turns like that can literally impact the effectiveness of your campaign like 200%, right.

Alexander Morisse: Yeah, 100%. Yeah, no, that stuff is super interesting, and also, you know, this sort of runs into this notion of a contextual bandit model, right, which is, how you would think about where do you play in the slot machine, right, and so in the beginning you have no idea what you'd have — no good hypothesis on which one you should play, but as soon as you start seeing some of the patterns, if you have enough time to play it 15 times each, right, you start getting a sense, "oh, this one here might have a slightly higher probability." Within this same vein, you sort of can understand, using prior knowledge, that it was a good idea to send the buy-one-get-one-free to people that are spending in some sort of segment that's lower, right, that's a big value-add, in other areas where people might be doing the $200 purchase, right, 30% off is a much better situation at that point.

Contextual Bandits and Ad Optimization

Alexander Morisse: So yeah, so that stuff is very interesting. Same thing with algorithmic ad buying, right, so you can do that exact same thing, you sort of sample the water, do one iteration, you update your priors, you update your hypothesis on what works and what doesn't, and then you continue that way, and so, after a short amount of time, you have sort of peaks in the space of where you should be putting your attention, and then also putting your ad spend.

Jake Litke: Yeah, well, let's dig into the multi-armed bandit thing, because I think that's a good analogy for people to wrap their head around, right, so you run an ad campaign, and you run a million impressions out to market.

The Multi-Armed Bandit Explained

Jake Litke: The multi-arm bandit, you know, is a methodology, but it's pretty easy to explain, so what you've done is you think of a slot machine, right, and you've pulled the lever now a million times, okay, and each time that you pull the lever, you got a different set of results, each individual pull has almost no value, right, but in aggregate —

Alexander Morisse: Yeah, that's where it becomes interesting.

Jake Litke: Yeah, so the idea is, right, let's say I have 15 slot machines, and I notice that baseline conversion is 1%, just hypothetical, but then you see that there's one that's 2.3%, right, obviously you want to go to that one. And so that's the idea, is that you sort of do this painful sampling that might cost a little burn-in, might cost a little money to do, but after that you know exactly where you should be hitting, and that can then translate to quite a bit of revenue.

Alexander Morisse: Yeah, we see — I mean that's the other thing about when you're doing programmatic and digital advertising, you're going to have, let's even just talk about clicks, right, which is not the best measure of attribution, but it's something that people understand — let's say you've got like a 0.1% CTR, which is somewhat average for digital programmatic, that means that for every thousand ads, one person is clicking on it, right, but you still served a thousand ads, right, and that signals out there.

Jake Litke: Mhm.

Finding the "Vein of Gold" in Ad Data

Alexander Morisse: So we see, like, and each individual impression doesn't cost very much, right, fractions of a penny for an individual impression, but if you start to look at — and these are like real numbers — the same ad campaign, if you look at the specific geography, so let's go down to the zip code of where the ads are being served and where purchases are happening, again, the publisher, which type of publisher is it on, how well does that resonate with your brand identity, right, because, let's say you have a health and wellness brand, it's very different from a lifestyle recreational brand, and those are going to function differently in different contexts. But looking at the numbers, you can see a specific geography, a specific time of day, a specific publisher, will be like a thousand percent better return than others, right, and that's why you need to dig into the data, because if you can find those — the cost of, like, you know, you serve 10 impressions —

Jake Litke: It's like a vein of gold inside of a —

Alexander Morisse: Exactly, like it's a vein of gold, but it's across the dimensions of time and context and location, and if you can figure out how to do that correctly, it's kind of like the stock market, like if you can buy low, sell high, that's where you can really drive a lot of profit.

Lessons from Building Beats Music

Alexander Morisse: Yeah, I mean it's the same thing, I mean, so the first time I really solved this — well, the first time I really, really solved this type of problem was when I was building Beats Music, right, which became Apple Music, and I was heading up the entire AI stack there, with all the recommendation scenarios for the music recommendation. And one of the things that we wanted to do was build a contextually relevant playlist generator that kept people engaged. And when I say contextually relevant, right, it really was in the context, like, we had to take into account day part — was it morning, midday, night, late night, right, people listen to very different music during those times. Were they traveling? Right, because we actually had location data when people were using the app, right, so if I make a business trip to New York, do I want to listen to the exact same music that I listen to when I'm home at the gym? Likewise, and, actually, yeah, so there was a lot of stuff there that we did, that was based around very simple heuristics, but then also got more and more complicated, right. So the simple heuristic was that, given the years that you were in high school, that was one of the biggest predictors on what type of music you were going to listen to, a lot of people get stuck in whatever music they listened to in high school, that's pretty common.

Jake Litke: Yeah.

Alexander Morisse: And so there was a lot of things that were kind of easy, low-hanging fruit, but then there was also the issue of, how do we get signal from the fact that — did they, you know, when they skipped a song, is that a negative signal, is that "I don't want to listen to this now," right, so we did all these types of AB tests there, where we would see that type of skip signal, and then we'd say, okay, put that back in at 10 songs later, right. One other thing I should mention that we found, is that people like repetition, right, so putting in constantly new music that was contextually relevant, in the same genre or whatever it was, was a huge turnoff to people, people like to hear the same song that they listened to 10, 20 songs later. And so, we did all this type of modeling that is actually completely analogous to what we're doing here in the ad world, right, so — yeah, I don't know what — why I went on that vein, but the notion that you have to do a lot of hypothesis testing on whatever signal is coming back from the device, or the graph, right, the pattern-of-life graph of the user, you have to do a lot of studies like this, and one of the simplest ways is to just utilize this Bayesian type of analysis, because it can very quickly show you the little peaks that are relevant to revenue.

Marketing as Emotional Storytelling

Jake Litke: Yeah, I mean, advertising, ultimately, in a — let's say a somewhat candid description — is like mind control, right, that's kind of what you're trying to — you're trying to influence people's minds.

Alexander Morisse: Yes.

Jake Litke: And let's just assume that we're doing this for generally good purposes, you have a product you believe in, you want people to know about it, you need to insert the awareness of your brand into their psyche, and then you want to drive them through the consumer journey, or the buyer journey, to become your customer.

Alexander Morisse: Yeah, I mean, this is the reason why think tanks in DC are populated with people that did marketing, right, they understand this exact same thing, repetition is one way to drive awareness of something, fear is another way, or FOMO, right, and, of course, all the other aspects — well, you know, sex.

Jake Litke: Well, yeah, and on that, I mean, most buying decisions come down to some sort of emotional decision, whether it's "I'm hungry and I want pizza," or "I am afraid, so I'm going to buy an alarm." And I don't think people in the new generation know this, right, but ad spend, or commercials and stuff back in the 50s, were a long laundry list of product descriptions and utility, right, and they realized this was a ridiculous way to sell things, right, and so, as early as the 20s, 30s, right, and so, once people understood that, they turned it into, what's the feeling that people want to feel, because that's going to have a much higher resonance in brand awareness than a list of features.

Alexander Morisse: Yeah, I like to use the example of insurance, right, because insurance is a boring topic, it does have — there's sort of a fear and protection emotional thing going on with it, "I'm already not listening" — so, yeah, but look at insurance ads, they're all humor, right, because that's kind of what they have to go with, they're like, "I'm gonna make you laugh," that's because I can't really talk to you about insurance because that's super boring, right. So successful brands, even when they're selling something that is, in some ways, unemotional or uninteresting, you have to figure out some sort of emotion to connect with, right.

Echo Chambers and Psychographic Clustering

Jake Litke: Yeah, yeah, it's interesting being aware of that, right, so once you've seen sort of the side of the industry where you have a view that most people don't have, like, who has sat around with 18 million impressions per day and analyzed it, right, very few people on the planet know what that looks like.

Alexander Morisse: Yeah, when you do that, segmented by geo, segmented by all the things that exist in the database, like political affiliation or all these things, you very clearly see huge differences in what is considered good or bad, right, or what actually drives click-through, right. So another company that I built was — well, now it's called Zeta Global, but it was Boom Train — and there we served basically all the native ads for CBS, Huffington Post, all the different news channels. And when I looked at that data, and I did clustering, regression modeling on likes, on things that we could get from third parties, it was very clear that people engage in echo-chamber type behavior, because they have a worldview that they hold on to and want to cling to. And for me this was an incredibly painful process, right, because I'd come from the academic sector, I believed very much that the academic world was the source of knowledge and truth and everything holy, and I realized that, no, in fact, most professors are highly propagandized people that aren't aware of it, right, and they're consuming a very small sliver of content that actually goes against any sort of rational, real, Bayesian reasoning. So, of course, I lost a lot of friends and stuff telling them, hey, you know what, you — don't tell anybody that their worldview is wrong, right, that's the worst thing you can do.

Jake Litke: Well, you can bring it back to emotion too, because when you tell human beings information that doesn't align with their worldview, that creates emotional discomfort, right, because you're now forcing someone to question things that they don't — people don't like that, right, most people don't like that.

Alexander Morisse: And I totally have empathy, because I didn't like it either, I really, really thought that the academic community was the most informed community on the planet.

Jake Litke: Yeah, and it was painful. All right, so let's see if we can reel it back in here.

Alexander Morisse: Pretty good. Not good, this was a sociographic exploration.

Jake Litke: It is helpful to understand, because again, there's, like, when people come to marketing that haven't spent a lot of time in it, and I keep saying this, they think that it's, you serve an ad and someone buys something because it's a pretty picture, or something that they want, but really it's about influencing mental state, it's about telling a story, ultimately all marketing is telling a story of some sort, whether it's humor or fear, but that's because that's how the human brain works, and that then correlates to emotion, right. So I think understanding that helps marketers make better decisions when they're trying to get their product out in the world. And another thing that people come to is, especially if you started a company, you believe in your product, you're like, "it's great, if I just tell people it exists, then they'll buy it," right, but the reality is they don't know anything about it, like, they don't know whether it's good, in some cases, what even it does or is. We've seen people forget to just put their name in their ads, sometimes they have the whole thing, and I'm like, "where's — how do they find you?" "Well, they're going to click on it." "Well, maybe they're not going to click on it, right." So now you're spending money telling people about something, but not even giving them a way to understand how to reach you, right. And it seems silly, but some of these fundamentals are where people get tripped up with the performance of their campaigns, because they get a little myopic about, like, "oh, it's 20%, or it's 30%," and you have to look at the broader picture to understand what's happening, so you don't waste your money, because that's the other thing about advertising, it is a terrific way to set your money on fire and waste it, right, the whole system.

Applying the Data to Clients

Alexander Morisse: So Jake, let me ask you this then, like, given what I know about the MediaJel dashboard, all the cool data science we're doing there and all that stuff, like when you're communicating this to clients, is it just like, "hey, at the end of the day, we can have a coefficient that's higher than one on what you spent versus what you're actually going to get"? What's the best way to communicate that to the average client?

Jake Litke: Yeah, so generally we're looking at purchases made, right, so we look at the purchases coming off the e-commerce, and then we tie the signal back to the impressions, and we have taken the approach of doing it in a transparent manner, right, so we actually give our advertisers all the data that we have. And one of the reasons that we do that is because people have very different opinions about what an appropriate attribution window is, is it one day, is it 30 days, and then how are you going to measure sort of the decay? Well, let's talk about that a little bit, you've got kind of a time decay on the signal, an impression that happened 30 days between the impression and the purchase, that's a weaker signal, right, and it's not really a linear reduction either, right.

Alexander Morisse: Yeah, I mean, so, in the realm of — sorry if I'm cutting off —

Jake Litke: No, no, go.

The U-Shaped Distribution of Attribution

Alexander Morisse: I was — in the realm of mathematics, like if we talked about this, right, there are distributions that we use, right, the Dirichlet distribution, right, negative binomial distribution, and what's amazing about these distributions is that they actually have these things where they're peaked on one end, very flat in the middle, and then peaked on the other end. So it can actually be that you're driving impressions for an entire month, right, and then you even just stop, right, you stop for two weeks, and then they see it one more time again, and now suddenly there's a decision to buy, right, and so that is a very well-known distribution, right, or it's a very known parameter space in that distribution.

Jake Litke: I just call it U-shape, is there a more scientific name for that?

Alexander Morisse: Yeah, I mean, it's the Dirichlet distribution, right, so, yeah, the Dirichlet.

Jake Litke: Uh, yeah, I don't know that one name.

Alexander Morisse: Yeah, that's the name. Yeah, and there's like, a gamma prior to that distribution, and it can have everything from a standard Gaussian, right, which is the sort of standard thing that everybody remembers, the mean, and then there's sort of a variance around —

Jake Litke: Right, the bell curve.

Alexander Morisse: But then it also has these places where you can go up into the space, and it basically has —

Jake Litke: Oh yeah, U-shape.

Alexander Morisse: Exactly, right, so it's sort of like this U-shape, and so there's a lot of things that are potentially happening low down, and then there's this sort of desert of nothing, and then suddenly there's a signal on the other end, right, so, very useful distribution for this type of modeling, and a lot of the stuff that you do in, like, lifetime value analysis, retention analysis, is using that type of prior.

Jake Litke: Yeah, when you start a campaign and you start measuring attribution, you get, day one, you're running impressions, and there's no sales, right, because it takes a couple days for that — so you generally look — it's a pretty standard-looking chart, kind of goes up and to the right, starts at zero, goes up, and then it'll start to level off, and you'll have sort of peaks and valleys, and you'll see, there's — it's nice when the data just makes sense, like, if you take a dispensary, for example, if you see the sales chart, like, Friday always has more sales, Thursday and Friday, like, pretty much consistently across the board, that's a busy day, right, so when you zoom out on a chart, and you're running for 6 months, you see every Friday, right, you see all the bumps, right, that makes sense.

Alexander Morisse: Totally.

Learning From the Data: Tuesdays and Taco Nights

Jake Litke: And then, you know, 4/20 happens and it goes off the charts and comes back, but in between Fridays, you still have variability, right, like, this week was better than that week, what's happening on Tuesdays? And we've been able to use this data to help people kind of level out their sales in some ways, right, so this is classic, like, bars, right, they have taco Tuesdays, why do they have taco Tuesdays? Because Tuesday is not like the normal day that people think of going out, so offer them free tacos and then they come in, right. And I think cannabis marketers can learn a lot just from looking at what's effective, and a lot of times look at yourself, like, what are the decisions that you're making that you didn't — how are you being influenced? And, you know, there's — well, hopefully we don't go too far off on a tangent here, but a lot of people say, "my phone is listening to me," right, which — let's just assume that that's not true for this conversation, okay, it may be happening, but let's just assume it's not — what's actually happening is that you give these social networks enough signal about yourself on a daily basis that they can pretty much predict what you're going to do in the future before you know you're going to do it.

Alexander Morisse: Yeah, and this is the spooky thing, right, like, I've worked in a ton of different contexts, I've never ever gotten a data source where the phone is listening, like, and I've seen a ton of stuff where that is pretty scary, but I've never seen that one, but it is really true, right, so there's this notion of latent variable modeling, right, so, if I show you a zoomed-up picture of the thread on a tire, right, you know right away it's a tire, right, there's no need to see the entire tire or even the hubcap, right, you just know. And so the same thing with a lot of these attribution, or this sort of like user behavioral analysis, is that they just need to see a very small snippet of something, and then they can basically infer pattern of life, or they can infer what cluster you're in, essentially, right, and that cluster will be people that ride motorcycles, that also love sushi, high-end sushi, blah blah blah, right — now you're just talking about yourself, what, now you're just talking about yourself.

Jake Litke: I assume you like sushi, I know you have a motorcycle.

Alexander Morisse: Yeah, so, yeah, I mean, there's nothing worse than low-grade sushi, so yeah, I was just — I had the best sushi ever, because I was in Japan recently.

Jake Litke: Yeah, right off the boat, and what was it, was it the butterfish, or —

Alexander Morisse: No, I mean, it was — I had salmon and toro, you know, just right off the boat, it's great.

Falling Into Buckets

Alexander Morisse: So yeah, that's the other thing that I think — another sort of human tendency is to believe that you are individually very unique, but most people are not that unique, you fall into specific buckets.

Jake Litke: You fall into buckets.

Alexander Morisse: Exactly.

Jake Litke: Yeah, I think I had that realization a long time ago, when I was reading Wired magazine, and I was like, this whole magazine, I like everything in this magazine, I'm like, but they give this to many other people, yeah, that also like everything in this magazine, so I'm not quite as unique as I might have wished I was.

Alexander Morisse: Yeah, I mean, this goes back to my previous comment about Boom Train or Zeta Global, like, it was bizarre to see my pattern of life in cluster number two, right, and then there was, like, 40 other clusters, and I was like, how can these people be living in the same reality, right, they're completely consuming different news, they have completely different views, etc, etc, but as I explore these different things, I actually — it's a sociographic analysis, like it's a user behavioral, psychographic analysis, you see very distinct archetypes emerge, right, and this is what Jung and Freud and everybody was talking about, I mean, Jung especially, from the archetypal perspective, from a slightly different perspective, but you saw this very clear — they postulate that there's these very clear signals in our psyche that drive behavior, and depending on what ratio of these things are expressed, you sort of fall into these different groups. So yeah, I mean, it's bizarre to imagine — I think we are very unique as individuals, in many respects, but there is this very clear notion of the mean, and whatever the mean is that you fall into, this is what you call culture, this is what you call society.

Brand Archetypes and Knowing Your Audience

Jake Litke: Yeah, and brands have their own archetypes, right, this is something that any major brand — and you can just look it up — they all are, are you the Sage, are you the Joker, are you the — whatever, Magician?

Alexander Morisse: Yeah, those things.

Jake Litke: And, you know, people like to — there's some pretty, like, Harley-Davidson is the Rebel, right. And, we're a little off topic, but understanding what your brand archetype is, is probably the first thing you need to do, because that matches very much with who you should be advertising to, and what words you should use when you communicate with them, right. If you get that — that's actually another thing, just like things to not do, again, a lot of — we deal with a lot of people that are newer businesses, and you're like, "okay, who do you want to advertise to?" And they're like, "everyone."

Alexander Morisse: Like that is —

Jake Litke: Yeah, exactly the wrong answer, right, you — first of all, you can't, because it costs too much money.

Alexander Morisse: Yeah.

Jake Litke: Second of all, even if you do try to go broad, it's just not going to work well for you, right.

Latent Variables and the Marble Analogy

Alexander Morisse: And let me do a quick introduction here on why this is interesting from the perspective of, like, Tableau users, or people that have something that is doing analysis over a database, right, so in your database you might have things like age, gender, whatever it is, and then you might be doing things like, "I want to look at females who are 18 to 23," right, and what's happening under the hood there is — think of everybody there as a colored marble, right, and now you're lumping all these colored marbles together, and then you're showing them something, but only the blue ones care, right, or only the orange ones care. And if you can actually not slice the data by age and all that stuff, but actually just identify this other property, this latent property that doesn't shine through from the database rows-and-columns perspective, but lives in the data through all these other things that you can imagine being latent variables in the data — if you can capture that latent variable, then you can determine, "oh, orange is the one I should go for," right, and then, boom, you see a huge, 40% uptick in revenue and sales.

Jake Litke: Yeah, you could also say that those marbles are made out of different substances, we'll go back to our gold analogy, some of those marbles are made out of gold, right, in terms of what that potential customer is going to do for you, from an initial sale, but from a lifetime value perspective, if you can build a strong brand affinity — I mean, you have to have your brand image, and then you have to deliver a good service or product, but that one, or 10, impressions that you paid 2 cents for, that got that person to be a customer, and then you delivered well for them, that person could be worth potentially many thousands of dollars to you, right. So it actually is the value of a little thing of gold, if you can identify which ones you should be targeting, and you really only can do that by running data, you've got to run advertising, collect the data, and you have to do so in a way where you're setting up — just like you said before, like, what is the actual copy, content, size, that you — what is the offer, the offers have to be linear independent, right, because if you get a user trying to gain the offer — wait a minute, if I spend a little bit more here and I can — right, then you're actually losing, right. So you want all the offers to actually be linearly independent, very independent of each other, and then you want to be able to understand that all these independent offers come with different segments, they come with very clearly defined segments.

Creative Testing: The Lotion Jar Example

Jake Litke: Yeah, and you can learn — I mean, like, I always learn interesting surprising things when you dig into this data. We had a campaign, it was for a topical product, like a lotion, there's two versions of the creative, one had just the jar, the other one had a hand holding the jar, right, the hand holding the jar performed twice as well as the other one did.

Alexander Morisse: Yeah, and there could be a number of reasons for that, but —

Jake Litke: Why I think one of them was — you could actually see how big the jar was, relative to the size of a hand, right, so then the consumer understands that they're not buying something big, they're buying something this big, and then the value is being conveyed through the imagery, right, so there's a lot of pieces to effective marketing. But you will learn a lot from the data, it will tell you when you're wrong too, because we have a good team, and we come up with ideas for targeting, and it works, we're pretty good at it, but we still get surprised sometimes.

Alexander Morisse: Well, that's the beauty of actually doing the real data science behind the scenes, right, because you actually have a mechanism there that can tell you if your hypothesis is wrong or not, and that is the essence of learning, it's the essence of exploration, and it's what gets you there.

Case Study: Texas vs. South Carolina

Jake Litke: Yeah, I have one last example — we have an advertiser that sells online, they don't have physical stores, they're just doing e-commerce, and Texas is their biggest market, right, by sales volume. When we ran detailed analysis, what we found was that, yes, Texas is where you're making lots of money, however, South Carolina, for the number of impressions purchased versus the number of dollars generated, is actually a four times better place to spend advertising — not that they're not going to run ads profitably in Texas, that's the mean — but you can see, like, here, I didn't even really think about that state, but it turns out that whoever is in that state, and your product alignment, is working fantastic, and you should spend more money in that state, right.

Alexander Morisse: Amazing.

Closing Remarks

Jake Litke: Yeah, so, all right, well, we're up on time here, that kind of went by quickly, hopefully we didn't get too nerdy or tangential.

Alexander Morisse: Well, I mean, I think we need to do another one where I bring my blackboard, right, so we can actually —

Jake Litke: And then maybe even do a quiz, yeah, we'll send a quiz, and if you can get any questions right, after — from Professor Alex, then we'll send out a prize.

Alexander Morisse: Well, this was great, good fun, and thanks for having me, and thanks to everybody who's listening.

Jake Litke: Yeah, and if people want to just maybe talk a little bit more about what you're doing at Louie, in case people are interested in getting in contact with you, why would they want to talk to you about that? I think it's interesting, but we didn't really talk about it.

Alexander Morisse: Oh, okay, great, yeah, no, that's fantastic, yeah, I mean, check out Louie.ai, just L-O-U-I-E dot A-I, and we're primarily going after the cybersecurity use case, but honestly, it's a general purpose platform, you can do genomics analysis in it, you can do all this pattern-of-life, you can do lifetime value analysis, anything that a business user would care about, where you need to do dashboards, interactive dashboards, interweave code and analysis, all using natural language — check out Louie, it's really, really a fun product.

Jake Litke: All right, thanks for that, Alex, appreciate your time, and we'll chat.

‍