Is Marketing Mix Modelling the future of B2B analytics? - Mark Stouse

Justin Norris: 00:00:14

Every marketing team wants attribution.

2

: 00:00:17

But the weird thing is that it's often not

that satisfying when they actually get it.

3

: 00:00:21

I did a lot of multi-touch attribution

projects as a consultant, and we got

4

: 00:00:25

really good at implementing these

tools, technically creating taxonomies,

5

: 00:00:29

making sure that data was clean.

6

: 00:00:31

But I found that when you actually show

these reports to a c-level executive,

7

: 00:00:36

it can be kind of underwhelming.

8

: 00:00:37

They don't always pass

the common sense test.

9

: 00:00:40

People want to pick on details, and maybe

that's because the idea of dividing up an

10

: 00:00:45

opportunity like a pizza between different

touch points isn't actually the way.

11

: 00:00:49

Maybe that's not going to

give us the answers we need,

12

: 00:00:53

but where does that leave us?

13

: 00:00:54

Because we still need to know what's

working in marketing, and today's guest

14

: 00:00:58

thinks there's a better way, and he's

founded a company to make marketing mix

15

: 00:01:02

modeling or MMM available to B2B marketers

and he's gonna tell us all about it.

16

: 00:01:07

So I'm super excited to

welcome Mark Stouse, CEO of

17

: 00:01:10

proof analytics to the show.

18

: 00:01:12

Thanks a lot for being here, mark

19

: 00:01:14

Mark Stouse: Hey, thank you so much.

20

: 00:01:15

Justin Norris: Mark, maybe

one clarifying question.

21

: 00:01:17

I've seen MMM, spelled out as media mix

modeling and marketing mix modeling, which

22

: 00:01:21

is the right way from your point of view?

23

: 00:01:24

Mark Stouse: actually.

24

: 00:01:24

It really represents the evolution of

it over the last say, 40 to 45 years.

25

: 00:01:31

Back when Procter and Gamble first brought

out what was then called econometric, I.

26

: 00:01:37

Analysis it was advertising.

27

: 00:01:40

that's what it was really all about,

hence the media mix modeling reference.

28

: 00:01:45

As, time moved on channels proliferated,

it became still very much within

29

: 00:01:51

kind of B2C a reference point.

30

: 00:01:54

It became marketing mix modeling.

31

: 00:01:57

Today it is really go to market mix

modeling because it includes not

32

: 00:02:04

just marketing data and channels and

investments and all that kinda stuff, but.

33

: 00:02:10

Sales and customer success and product

data outside data externalities, you

34

: 00:02:17

know, the economy, your competitor

actions, really is today much

35

: 00:02:22

broader canvas as it should be.

36

: 00:02:25

Right?

37

: 00:02:26

That's the long and the short of why

people say that MMM means media mix

38

: 00:02:33

modeling or marketing mix modeling.

39

: 00:02:35

And actually both of 'em are

kind of A little bit passe today

40

: 00:02:39

Justin Norris: Maybe let's take a

step back before that and talk about

41

: 00:02:43

what was your professional experience

leading up to founding proof analytics

42

: 00:02:46

and what brought you to this direction?

43

: 00:02:49

I.

44

: 00:02:49

Mark Stouse: So I was actually pretty

much a classic marketer and communicator.

45

: 00:02:54

I've worked across all of the different

subsets of marketing at one time or

46

: 00:02:59

another, and I've been a large company,

CMO, . About a little less than 20

47

: 00:03:04

years ago I was at HP and we were all

kind of in the middle of an existential

48

: 00:03:09

crisis because the then CEO of hp,

mark Hurd was a very Operations focused

49

: 00:03:17

and a very customer focused CEO.

50

: 00:03:20

And he wanted to know why there wasn't

more evidence of what marketing was

51

: 00:03:27

actually delivering to the company.

52

: 00:03:29

It was actually incredibly unpleasant.

53

: 00:03:32

And about the only good thing that I can

say about that whole experience that I had

54

: 00:03:36

and that other peers of mine had was that

it was, was highly motivating to change.

55

: 00:03:44

I kind of got to a point where I

said to myself, look, I either have

56

: 00:03:48

to do something to fix this or I

just need to like go do something

57

: 00:03:53

else, 'cause it's not just about

budget issues and all that kinda stuff.

58

: 00:03:58

It's about credibility,

am I actually doing here?

59

: 00:04:01

So in my particular case for whatever

reason, herd gave me a project.

60

: 00:04:09

To actually see what could

be done to resolve this.

61

: 00:04:13

And I didn't know Jack at

that time about analytics.

62

: 00:04:19

I was not even a math enthusiast,

and and so I, I remember I went

63

: 00:04:24

home that next Friday I went

through all the stuff in my garage.

64

: 00:04:30

. And found a couple of old math

textbooks from university and started

65

: 00:04:36

to read them in ways that I never

read them when I was in school.

66

: 00:04:40

' cause I was actively seeking an answer

and all of a sudden I got to this and it

67

: 00:04:45

was all about multi-variable regression.

68

: 00:04:49

Which is absolutely the cornerstone of

causal analytics ? It is the cornerstone

69

: 00:04:56

of the scientific method of inquiry.

70

: 00:04:59

It has so much credibility that,

it'd be hard to have more credibility

71

: 00:05:04

than multi-variable regression has.

72

: 00:05:07

Not because it is perfect, but it

is absolutely the best that we have.

73

: 00:05:13

And today, is actually

the bedrock of causal ai.

74

: 00:05:18

So I started working on this project

for Herd, and he liked it a lot.

75

: 00:05:26

And I guess my prize was he

set up a mentoring session.

76

: 00:05:32

With the CFO of hp, Bob Leman.

77

: 00:05:35

The whole idea was, I want you guys to

talk you from your end and he from his end

78

: 00:05:42

and see if we can't get to a meeting of

the mines between marketing and finance.

79

: 00:05:48

This whole experience it just changed me.

80

: 00:05:51

And before he passed away, I had

the opportunity to talk to Mark

81

: 00:05:55

again and thank him for it, right.

82

: 00:05:57

Because it was, not fun at all,

but he did me the hugest favor.

83

: 00:06:03

So I started climbing the

analytics ladder, right?

84

: 00:06:08

The stairway to heaven, so to speak.

85

: 00:06:10

Right?

86

: 00:06:10

By the time I was CMO at Honeywell

aerospace, pre any kind of

87

: 00:06:15

automation for analytics, right?

88

: 00:06:17

meant we had to hire a ton of people

in order to get the latency on the

89

: 00:06:22

recalculation down, the scalability up

the cost be damned at this point, right?

90

: 00:06:30

'cause it was so important.

91

: 00:06:31

And get the understandability of

the outputs to the point where the

92

: 00:06:36

business goes, yeah, I not only get

it and believe it, but I can make a

93

: 00:06:40

better decision today than I could

make before as a result of this.

94

: 00:06:44

And then you kind of say, there's not very

many companies that are willing to invest

95

: 00:06:51

seven, eight, $9 million a year just in

marketing analytics particularly in B2B.

96

: 00:06:58

That point you start to realize

that, that automation was gonna

97

: 00:07:01

be absolutely indispensable.

98

: 00:07:04

To solving the underlying root problems,

if Data science has has a number

99

: 00:07:09

of major problems in the business

context, but they can kind of be summed

100

: 00:07:13

up by the fact that it's too slow.

101

: 00:07:17

So by the time they get you the

insights, everything has moved on,

102

: 00:07:20

you've already had to make the decision.

103

: 00:07:23

The information is not comprehensible

many times by normal people.

104

: 00:07:29

it's not scalable.

105

: 00:07:31

So you end up with three or four

mega models that get updated

106

: 00:07:35

once, maybe twice a year.

107

: 00:07:38

And so not agile, right?

108

: 00:07:40

But big opportunity to appropriately

automate and bring ai into that

109

: 00:07:47

whole thing and really elevate

all of this to a new level of

110

: 00:07:51

accessibility and meaningfulness.

111

: 00:07:54

And so that's why we built proof.

112

: 00:07:56

Justin Norris: And so I want to

contrast MMM and regression based

113

: 00:08:00

analytics that you're describing with.

114

: 00:08:02

What most B2B marketers are

probably familiar with, which is

115

: 00:08:06

multi-touch attribution, that's what

pretty much everybody does today.

116

: 00:08:10

What is the problem with this form of

attribution from your point of view?

117

: 00:08:14

Or is there a problem?

118

: 00:08:15

Is it still okay?

119

: 00:08:15

Can they be complimentary?

120

: 00:08:17

Mark Stouse: as long as it's accurate

data, there's absolutely nothing wrong

121

: 00:08:21

with having a really good understanding

of the customer journey, patterns in the

122

: 00:08:26

customer journey, things like that, right?

123

: 00:08:29

And indeed, a lot of our customers

will include customer journey

124

: 00:08:34

data in regression models.

125

: 00:08:36

It's valuable.

126

: 00:08:38

The problem is that it is not fit for

purpose for what it's being sold to do.

127

: 00:08:47

We can just start with the idea of bias.

128

: 00:08:50

There's a lot of bias in that data.

129

: 00:08:53

There's a lot of bias in

the weightings of that data.

130

: 00:08:57

So marketing teams set their

own weightings in advance.

131

: 00:09:02

It is antithetical to the mathematical

principles, to say that you can

132

: 00:09:07

use MTA data to optimize anything.

133

: 00:09:11

Particularly spend

134

: 00:09:13

Justin Norris: I just want to

unpack that to understand why

135

: 00:09:16

Mark Stouse: mTA data is an effect.

136

: 00:09:19

You're measuring an effect.

137

: 00:09:21

not capturing.

138

: 00:09:22

The cause of that.

139

: 00:09:24

And more specifically,

the time lagged cause.

140

: 00:09:28

So how are you going to optimize the

money that you spent the past at this

141

: 00:09:35

point, based on what you think you're

seeing from effects in the present when

142

: 00:09:41

you have no idea what the time lag is?

143

: 00:09:43

Right?

144

: 00:09:43

So there is this assumption,

that the time lag is near zero,

145

: 00:09:48

but that's just so not the case.

146

: 00:09:50

Mm-hmm.

147

: 00:09:51

I B2B, the time lags can be extensive.

148

: 00:09:54

What do I mean by extensive?

149

: 00:09:56

Two, three quarters.

150

: 00:09:58

we get into brand investments, it's

double that, exactly are you optimizing

151

: 00:10:03

and how do you know which part of your

budget back in time you needed to optimize

152

: 00:10:09

based on this effect in the present?

153

: 00:10:12

Right?

154

: 00:10:12

Justin Norris: If we take an MTA

scenario, just to make it like a

155

: 00:10:15

little bit more clear, let's say

we have an opportunity and we have

156

: 00:10:18

three people in the buying committee

that are part of that opportunity.

157

: 00:10:21

And between those three people,

there were sacred brown numbers, 30

158

: 00:10:24

different touch points that they had

with different marketing initiatives.

159

: 00:10:28

And those could be like, they

downloaded eBooks, they attended

160

: 00:10:30

webinars, they clicked on ads.

161

: 00:10:32

And so the, traditional way is like,

we're gonna take those 30 touchpoints

162

: 00:10:36

and we're gonna take that opportunity.

163

: 00:10:38

Let's say it's a 3 million opportunity,

and we're just gonna divvy it up.

164

: 00:10:41

And the way that we

divvy it up could vary.

165

: 00:10:43

We may pick some touch points, is being

more influential or not influential?

166

: 00:10:46

And I totally agree with you there,

that it's kind of like arbitrary that

167

: 00:10:49

they give you different models that

you can choose from and it's like, pick

168

: 00:10:52

the one you want, like on what basis?

169

: 00:10:56

Mark Stouse: I don't think

that marketers understand how

170

: 00:10:59

transparently untrue the premise of

MTA is to everyone else but them.

171

: 00:11:07

Justin Norris: I wanna get to the

reality of the situation because this

172

: 00:11:10

is the, the industry that we're in.

173

: 00:11:12

Mark Stouse: The reason why it

went through adoption is that most.

174

: 00:11:16

Marketers don't have enough knowledge

of math to be able to recognize

175

: 00:11:23

when something doesn't work.

176

: 00:11:24

And number two, it was presented to

them as a way of mining data that they

177

: 00:11:31

already were generating in their stacks.

178

: 00:11:34

So it was easy in quotes, right.

179

: 00:11:38

To do this.

180

: 00:11:39

And I've been around it long enough

and talked to enough marketers about

181

: 00:11:43

it, everybody was shocked when they

went into their first meeting with

182

: 00:11:50

MTA data, Thinking that they were

just gonna be greeted as conquerors

183

: 00:11:55

and victors and to your earlier

point at the top of the show, right.

184

: 00:11:59

It was anything but that.

185

: 00:12:01

it was very innervating

for a lot of marketers.

186

: 00:12:06

You can actually do it in

Excel if you only have to do

187

: 00:12:09

it on a very limited basis.

188

: 00:12:11

But if B2B would just adopt B2C analytics

and also I would just add market research

189

: 00:12:20

priorities they would see a very rapid

transformation of their credibility and

190

: 00:12:26

their precision and everything else.

191

: 00:12:28

And.

192

: 00:12:29

can always modify it as needed,

but the core principle is ready to

193

: 00:12:35

go and has been a very long time.

194

: 00:12:38

Justin Norris: So if I recap your

perspective as I understand it, touch

195

: 00:12:41

points, the customer journey data,

what the person did and when that can

196

: 00:12:44

be interesting, that can be valuable.

197

: 00:12:46

The notion of then doling out credit

to those touch points for revenue

198

: 00:12:51

outcomes, not mathematically,

statistically defensible.

199

: 00:12:55

Mark Stouse: would just add this, right?

200

: 00:12:56

MTA at the end of the day

is about pattern matching.

201

: 00:13:00

It's about identifying large

scale, repeating patterns

202

: 00:13:03

in the customer journey.

203

: 00:13:06

And then the assumption is that if a

bunch of people are doing the same thing

204

: 00:13:10

over and over again, it must be causal.

205

: 00:13:13

That is a complete fallacy, the

same math is used for example,

206

: 00:13:19

and in fact, pattern matching too

is used to study climate change.

207

: 00:13:24

Some are causal, some are

absolutely not causal.

208

: 00:13:30

They happen all the time,

but they don't mean anything.

209

: 00:13:35

You can't just assume that

because, a thousand people do it.

210

: 00:13:40

Okay.

211

: 00:13:41

One of your channels, one of your feedback

loops, you know, it just keeps happening

212

: 00:13:44

that ipso facto means that it's causal.

213

: 00:13:48

Justin Norris: I've always had this

unease as well where you a great

214

: 00:13:51

example from earlier in my career,

we had a self-serve SaaS app, what

215

: 00:13:55

would be called product led today.

216

: 00:13:57

And everybody that signed up for

that app got a welcome email.

217

: 00:14:01

So from that point of view, if you're

looking at touchpoints, like the

218

: 00:14:03

welcome email is very influential.

219

: 00:14:05

It's very important but,

but everybody got it.

220

: 00:14:07

So just because people got it and

some people opened it doesn't mean you

221

: 00:14:11

could necessarily infer that that was

driving the outcome in a particular way,

222

: 00:14:16

. So everything we've been discussing

is kinda like the current state.

223

: 00:14:20

Let's contrast it now with MMM and

why does it not have these different

224

: 00:14:26

shortcomings that we've described?

225

: 00:14:28

Mark Stouse: Number one, let's start with

the shortcomings that it does have, right?

226

: 00:14:31

It is very dependent, as is

every analytic of any kind.

227

: 00:14:37

Very dependent on data quality, There's

that old saying about gigo, right?

228

: 00:14:42

Garbage in, garbage out.

229

: 00:14:44

Everyone is going to be

victimized by gigo equally.

230

: 00:14:49

Beyond that though.

231

: 00:14:51

It has a lot of major advantages.

232

: 00:14:54

It is a lean data mathematical process.

233

: 00:14:57

So you do not have to have big data

or even a lot of lean data order

234

: 00:15:03

to run these models accurately.

235

: 00:15:07

And that is actually a huge factor.

236

: 00:15:09

One of the things that CDOs chief

digital officers or data officers are

237

: 00:15:13

grappling with right now on private

LLMs is that they big data stocks.

238

: 00:15:19

They don't have enough training

data and they don't have enough

239

: 00:15:22

operational data to run private LLMs.

240

: 00:15:26

Everyone's been so understandably

fascinated with what they can do

241

: 00:15:30

with public LLMs that they haven't

really thought it through in terms

242

: 00:15:36

of the limitations on private stuff.

243

: 00:15:39

Right?

244

: 00:15:40

regression does not have this problem.

245

: 00:15:42

If you say, oh, I don't have enough

data to run regression analytics.

246

: 00:15:47

not even remotely able

to do any kind of ai.

247

: 00:15:52

And I think that a lot of c-suites

now Are starting to say, your

248

: 00:15:56

creation and maintenance of a high

quality data pool that's relevant to

249

: 00:16:01

your function is a core competency.

250

: 00:16:05

And if you tell us that you don't have

the right data, you don't trust your

251

: 00:16:10

data that's on you, The other thing is

you have to have some basic capability

252

: 00:16:15

on the human side of the equation.

253

: 00:16:18

So with proof, for example, we have

simplified the whole thing, dramatically.

254

: 00:16:24

Automated it significantly

in the right places.

255

: 00:16:27

you don't need a full-blown

data scientist to run it.

256

: 00:16:32

You need a competent data

analyst and you really don't

257

: 00:16:35

even need that person full time.

258

: 00:16:38

That's probably a half FTE, right?

259

: 00:16:41

Justin Norris: Could we give a

Fisher Price version definition of

260

: 00:16:45

what MMM is like, just so people

can conceptualize it in their minds.

261

: 00:16:50

Mark Stouse: we live in a multi-variable

world there's tons of potential

262

: 00:16:55

causes everything that you control

and everything that you don't control.

263

: 00:16:59

That's kind of a really

super easy way to bucket it.

264

: 00:17:03

And these all have interactions with

each other across time and space

265

: 00:17:08

that produce particular outcomes.

266

: 00:17:11

Causally speaking this is a

probabilistic calculation.

267

: 00:17:15

So outside of certain physical laws like

gravity, There is no deterministic outcome

268

: 00:17:22

that's possible to determine, right?

269

: 00:17:25

You'd have to know everything

that there is to know about what

270

: 00:17:29

contributes to a particular outcome

to get to a deterministic answer.

271

: 00:17:34

That's just not the way that

operates, particularly when you're

272

: 00:17:37

talking about human behavior.

273

: 00:17:39

So this is the same math that's

being used routinely to study climate

274

: 00:17:46

change, to study epidemiology.

275

: 00:17:50

To study economics, it captures time lag.

276

: 00:17:54

So in the end, the report will tell

you historically the stack rank

277

: 00:17:59

of everything that you control and

don't control, that's in the model.

278

: 00:18:04

It's relative effect on this outcome.

279

: 00:18:07

It will then forecast all that

into the future so that you can

280

: 00:18:12

then make different choices.

281

: 00:18:13

And because of automation,

we've sped that up to the point

282

: 00:18:18

where you can run multi-variable

regression exactly like GPS.

283

: 00:18:23

say that you're recalculating the model

every week, new data comes in and is

284

: 00:18:29

presented to the model automatically,

it automatically recalculates.

285

: 00:18:34

And you see how the present now

is comparing with the forecast

286

: 00:18:39

that you have for the same period.

287

: 00:18:41

if there's a growing delta, right?

288

: 00:18:44

You see, hey, okay, I

need to make some changes.

289

: 00:18:46

Stick with the GPS analogy.

290

: 00:18:49

need to reroute.

291

: 00:18:51

Or man, this is great.

292

: 00:18:52

It's just totally tracking.

293

: 00:18:53

And it will tell you how long, longer

it's going to take for you to reach your

294

: 00:18:59

objective, your goal, your destination.

295

: 00:19:01

So there's a countdown.

296

: 00:19:03

Regression is a huge part of

the actual GPS that's on your

297

: 00:19:08

phone, you use every day.

298

: 00:19:10

And if you stop and think about

it for a second, most business

299

: 00:19:14

questions, and indeed a lot of life

questions are navigation questions.

300

: 00:19:19

Where am I?

301

: 00:19:20

Where do I want to go?

302

: 00:19:21

What's the best way to get there?

303

: 00:19:24

I have enough time and resources

to achieve my destination?

304

: 00:19:29

Am I gonna run outta

gas before I get there?

305

: 00:19:32

I gonna run out of time

before I get there?

306

: 00:19:34

Am if I have to be there at

nine o'clock for a meeting?

307

: 00:19:37

And the GPS says, you're

not getting there until 11.

308

: 00:19:40

gonna have to make some

choices, that's part of it.

309

: 00:19:44

also captures all the headwinds

and tailwinds that may be speeding

310

: 00:19:49

you up or slowing you down.

311

: 00:19:51

So in many ways, it it gives you

the answer to your questions.

312

: 00:19:56

Justin Norris: Take a practical example

along the lines of one of those questions,

313

: 00:19:59

let's say a company spends a hundred

thousand dollars a month on display

314

: 00:20:04

ads as an act of faith because those

can be notoriously difficult to track.

315

: 00:20:08

They're not always resulting

in a click, but they could have

316

: 00:20:11

impressions that could have an impact.

317

: 00:20:13

So they're spending that a hundred

K month over month, and the CFO

318

: 00:20:16

challenges the CMO and says, why.

319

: 00:20:19

Are we spending this 100 K every month?

320

: 00:20:21

What's it doing?

321

: 00:20:22

And that's the question

the CMOs trying to answer.

322

: 00:20:25

How would MMM, like what does it compare

that spend versus a particular outcome?

323

: 00:20:31

And try to show a causal

relationship between them.

324

: 00:20:34

Mark Stouse: yeah.

325

: 00:20:35

Multiple variables, right?

326

: 00:20:36

So it's highly context oriented.

327

: 00:20:39

All of these models are.

328

: 00:20:41

Seek to capture as much of

a known context as possible.

329

: 00:20:45

what a lot of marketing teams figure

out using exactly that scenario,

330

: 00:20:52

is they've been trying to justify

that display ad budget in terms of

331

: 00:20:58

demand gen, in terms of, performance

marketing, And that is just not the

332

: 00:21:04

way that that typically plays at all.

333

: 00:21:07

That is a brand reputation investment.

334

: 00:21:12

know, you kind of think about

marketing expense as being

335

: 00:21:15

essentially two big chunks, right?

336

: 00:21:18

Branded demand brand is easily two to

three times time lagged in its effect.

337

: 00:21:25

Than demand is.

338

: 00:21:27

It also sticks around a lot longer.

339

: 00:21:30

The effects of brand investment

doesn't deteriorate very quickly

340

: 00:21:35

unless there is a major scandal of some

sort, something like that, that all

341

: 00:21:40

of a sudden breaches the trust wall.

342

: 00:21:42

But absent that right, it hangs

around The halflife is quite extended,

343

: 00:21:47

whereas the Halflife on demand

investment is highly perishable.

344

: 00:21:52

Lasts maybe a couple months and

then it's gone, so the snows melt

345

: 00:21:58

a lot faster with demand than brand.

346

: 00:22:01

You have to then say to finance,

never discussed time lag with you

347

: 00:22:08

before specifically, but we all know.

348

: 00:22:11

That marketing takes

time to have an effect.

349

: 00:22:14

That simple statement is insufficient

because if we don't know the

350

: 00:22:20

time lag, we will never know.

351

: 00:22:22

The ROI, so we now have the ability

to say that this investment over

352

: 00:22:29

here doesn't really drive demand,

doesn't really pay off on this side.

353

: 00:22:35

There isn't a quick return a demand

perspective, but from a brand perspective,

354

: 00:22:42

this is how it is improving average

deal size and average deal velocity.

355

: 00:22:47

Those are the two big ones that we see

again and again on brand investment.

356

: 00:22:52

It is grease on the wheel of the deal.

357

: 00:22:55

That's what brand

reputation really is, right?

358

: 00:22:59

It makes people buy more than

they otherwise would and buy

359

: 00:23:04

faster than they otherwise would.

360

: 00:23:07

that That is key in B2B because

it's a higher cost, higher

361

: 00:23:12

risk by decision to begin with.

362

: 00:23:15

And then today, if we layer in all the

risk factors that people are worried

363

: 00:23:20

about it's even more important,?

364

: 00:23:22

I mean, If you really want to understand

how trusted you are by your customer

365

: 00:23:27

base, look at your average deal velocity.

366

: 00:23:30

If you really want to understand how

confident people are in your product's

367

: 00:23:35

ability to generate massive amounts of

value for them, look at their average

368

: 00:23:41

deal size, particularly year one, right?

369

: 00:23:43

Certainly year two, your, a

lot of people, just as a matter

370

: 00:23:46

principle today, are buying small

in year one and testing it out.

371

: 00:23:51

If you see a major uptick in

year two and deal size, that is a

372

: 00:23:59

major vote of confidence in you.

373

: 00:24:01

you don't, you need to

really find out why that is.

374

: 00:24:07

'cause I guarantee there's a reason.

375

: 00:24:09

Justin Norris: Looking at our

example, let's say the dataset that

376

: 00:24:12

you had, the company had always

been running those display ads.

377

: 00:24:16

Do you need a period where those display

ads are not there in order to demonstrate

378

: 00:24:20

the impact of having them or not?

379

: 00:24:22

Or is there a way, even though they've

always been there, to somehow demonstrate

380

: 00:24:27

the impact that they're having?

381

: 00:24:29

Mark Stouse: I think that the answer to

that question is very much contextual.

382

: 00:24:35

if we were talking about a correlation

analysis, meaning this ad buy right

383

: 00:24:40

against revenue, Then yeah, you would

definitely need some interruptions

384

: 00:24:45

to be able to essentially create a

385

: 00:24:48

Justin Norris: Oh, control, that's

the word I was looking for too.

386

: 00:24:50

Yep.

387

: 00:24:51

Mark Stouse: If you're talking about

multi-variable though, where a lot of

388

: 00:24:54

things are changing all the time around

it, it's not necessary that's actually

389

: 00:24:59

one of the things that's really, really

cool about MVR multi-variable regression

390

: 00:25:03

is that because you're bringing in so

many different variables and because

391

: 00:25:08

everything changes, particularly the

stuff that you don't control you're gonna

392

: 00:25:13

see exactly what you're talking about.

393

: 00:25:14

You're gonna see implicit AB test.

394

: 00:25:19

Evolving across time.

395

: 00:25:20

Justin Norris: We're

talking about limitations.

396

: 00:25:21

And one of the things we're talking

about was being an aggregate

397

: 00:25:25

versus tracking individual users.

398

: 00:25:28

Mark Stouse: Yeah, so one of the things

that's really important to say about

399

: 00:25:32

multivariable regression slash go to

market, mixed modeling, et cetera, right?

400

: 00:25:38

Is that it is looking at all the

causal relationships in aggregate.

401

: 00:25:44

So it's not possible to say that all

these things cause this one identified

402

: 00:25:52

person to take the steps that they took.

403

: 00:25:55

That's not what go to market

mix modeling is all about.

404

: 00:25:58

It's about looking at a population wide

trend that's causal for a lot of people

405

: 00:26:06

one of the things that companies really

appreciate though about this is that.

406

: 00:26:10

There's no PII in proof

or in, a regression model.

407

: 00:26:15

So you don't have the security

concerns that you would have in

408

: 00:26:20

MTA or any of the other touch based

approaches, which are trying to tie.

409

: 00:26:27

Broad investments, programmatic

investments to impact on a particular

410

: 00:26:33

human being or particular company.

411

: 00:26:35

And to do that, you have to

pierce the veil on identity.

412

: 00:26:39

And that's a problem.

413

: 00:26:41

Justin Norris: So it's

looking at the big picture.

414

: 00:26:42

So cookies, not an issue,

GDPR, not an issue.

415

: 00:26:47

It's really comparing

bulk data sets and the

416

: 00:26:50

Mark Stouse: In fact, the only GDPR

thing that we ever have to encounter is

417

: 00:26:57

the login information for users of proof.

418

: 00:27:00

So we keep that secured, But in

terms of the data that's in the

419

: 00:27:06

computations, it's a non factor.

420

: 00:27:09

It's just non issue.

421

: 00:27:11

Justin Norris: Is there a way to

explain that multi-variate regression

422

: 00:27:14

to someone who maybe they've got high

school math, first year university

423

: 00:27:17

math, 'cause maybe that's a challenge.

424

: 00:27:18

As you said, marketers, some of them

are more quantitatively oriented.

425

: 00:27:22

Not all of them are.

426

: 00:27:23

And they're gonna need to trust,

this data, so they'll need to be able

427

: 00:27:26

to make sense of how it's produced.

428

: 00:27:29

You know, If it's just a kind

of computer and it spits out an

429

: 00:27:31

answer like, yep, do X or do Y.

430

: 00:27:33

Mark Stouse: They're already effectively

thinking in this way because they're

431

: 00:27:38

spreading out their investments

across all kinds of different

432

: 00:27:40

channels, So they're acknowledging the

multi-variable world in which they live.

433

: 00:27:46

What they are not necessarily

acknowledging is the fact that it's not

434

: 00:27:52

a list of one-to-one correlations, My

investment in display ads versus revenue,

435

: 00:27:59

my investment in social versus revenue,

my investment here versus revenue.

436

: 00:28:03

That's not it.

437

: 00:28:05

It's a tapestry that weaves back and forth

with different time lags associated with

438

: 00:28:11

hmm.

439

: 00:28:11

Literally the only way capture that

is with multi-variable regression

440

: 00:28:16

or econometrics or, marketing

mix modeling, go to market mix

441

: 00:28:20

modeling, right there, it's all

442

: 00:28:21

the same thing,

443

: 00:28:22

Justin Norris: the fir, the

first thing that you mentioned

444

: 00:28:24

Is what we're conditioned to look for.

445

: 00:28:25

Like we were conditioned to say like, I

invested a dollar here and it produced

446

: 00:28:30

two, like these definitive statements.

447

: 00:28:33

I think if I'm interpreting you

correctly, you're saying reality is not

448

: 00:28:35

that straightforward most of the time.

449

: 00:28:37

Mark Stouse: The final output

can be that straightforward.

450

: 00:28:40

Okay?

451

: 00:28:41

But the causality elements on

it, are going to vary with the

452

: 00:28:45

wind, so a great example of this

is uh, Johnson Controls, right?

453

: 00:28:50

So even before Covid really became a

story, uh, their analytics started to

454

: 00:28:57

say in terms of forecasting, right?

455

: 00:28:59

for reasons we don't understand.

456

: 00:29:01

All these different channel investments

that we have historically made are

457

: 00:29:06

showing that they're not gonna be as

effective in six months as they are now.

458

: 00:29:11

A lot of these were physical events.

459

: 00:29:13

, this was not due to marketing

data in these models.

460

: 00:29:17

This was all about external data

was already up these early signals,

461

: 00:29:23

they decided, you know what?

462

: 00:29:24

We are gonna fly by instrument.

463

: 00:29:27

We're not going to fly by what we can see.

464

: 00:29:30

And so they started removing investments

early a lot of these things and they were

465

: 00:29:36

able to claw back quite a bit of money.

466

: 00:29:38

And then all of a sudden, real

life proved the analytics accurate.

467

: 00:29:44

And so they were very happy about that.

468

: 00:29:46

And then finance came later and said, Hey,

so sorry, we're gonna have to cut you.

469

: 00:29:53

30 or 40%.

470

: 00:29:54

I can't remember exactly

what it was, but it large.

471

: 00:29:56

And they said, let us show you something.

472

: 00:29:59

Right?

473

: 00:29:59

And they modeled the effects of those

kinds of cuts over the next three

474

: 00:30:05

years, and it was so detrimental

to the business that finance took

475

: 00:30:10

a lot less and went elsewhere to

get whatever they needed, right?

476

: 00:30:15

So this is an example, not only of

how a marketing team used it to avoid

477

: 00:30:22

waste, but also to avoid cuts that would

be ultimately bad for the business,

478

: 00:30:28

Justin Norris: and the sort of external

data you just described, those are things

479

: 00:30:31

that's not specific to any one business.

480

: 00:30:33

It's common to the macro environment

that many businesses are working in.

481

: 00:30:39

Like if I come to you, do I need to

bring that external data on my own?

482

: 00:30:42

Or do you have sort of global

external data that you can match

483

: 00:30:46

up with my business specific

data and include in the model?

484

: 00:30:50

Mark Stouse: So we are not a

data provider, but we can and do

485

: 00:30:55

all the time help people locate

data sets that are usually free.

486

: 00:31:02

That kind of stuff is usually

free, either from the government

487

: 00:31:06

or major financial institutions

or universities, things like that.

488

: 00:31:10

So that's not a problem.

489

: 00:31:12

And it's great.

490

: 00:31:13

Usually just fantastic data.

491

: 00:31:15

I mean, It's been totally scrubbed,

so that's not a big deal at all.

492

: 00:31:20

In fact, we live in the golden

age of data availability.

493

: 00:31:25

Even if it's not free,

someone is measuring it.

494

: 00:31:29

Purely speculatively in the belief

that somebody is gonna wanna buy it.

495

: 00:31:34

And so today with a credit card,

you can subscribe to all kinds of

496

: 00:31:39

data sources, cost effectively.

497

: 00:31:42

Particularly given how important

it is to maximizing the investment.

498

: 00:31:48

The other thing that I would

just point out is ROI is really

499

: 00:31:52

important, but you only know RROI.

500

: 00:31:55

Looking backwards, right?

501

: 00:31:57

ROI is a historical assessment.

502

: 00:32:00

What is really important is

forecasted, ROI, and then the

503

: 00:32:05

comparison between it and actuals.

504

: 00:32:08

So again, this is very much like

public companies issue guidance

505

: 00:32:13

and then they issue regular

updates against that guidance.

506

: 00:32:17

That is exactly what the C-suites of many

companies are, demanding today, right?

507

: 00:32:23

It's like an investment

uh, to start a new company.

508

: 00:32:27

The first question the investor's

gonna ask is, what do you expect

509

: 00:32:30

this to do in year 1, 2, 3, 4, 5?

510

: 00:32:33

And what's the basis for that projection?

511

: 00:32:37

If you're just extrapolating from hope

and best wishes and all that kind of

512

: 00:32:42

stuff, that's not much of an argument.

513

: 00:32:45

But if you're doing regression based

analysis, that starts to mean something.

514

: 00:32:51

Justin Norris: And listening to the

examples that you described, it does feel

515

: 00:32:54

like a very enterprise oriented solution.

516

: 00:32:57

If I work at a company, 400

people, 50 million a RR, is this

517

: 00:33:01

something that can work for us, or

do I need to be of a certain size

518

: 00:33:04

threshold for it to be useful?

519

: 00:33:07

Mark Stouse: No, actually

it totally scales.

520

: 00:33:09

The reasons for investing in it are

going to be different in a small,

521

: 00:33:14

medium, or enterprise type business.

522

: 00:33:17

But it totally scales.

523

: 00:33:19

And the cost is totally approachable,

even for mom and pop, right?

524

: 00:33:24

So the reasons for doing it as

a small, and let's say the lower

525

: 00:33:28

half of the medium sized business.

526

: 00:33:30

Are mainly because if you make a bad

investment in whatever, It has a almost

527

: 00:33:37

immediately negative effect on cashflow.

528

: 00:33:41

Too much risk, so they are

modeling to avoid that.

529

: 00:33:45

We do have some small customers and

that's their main reason for doing it.

530

: 00:33:50

If you are an enterprise and

you're spending, I don't know, $200

531

: 00:33:53

million a year on marketing, right?

532

: 00:33:56

you spend it wrong, what the CFO

is most concerned about is things

533

: 00:34:03

like opportunity cost, right?

534

: 00:34:05

That's the comparison that's going on

in their mind all the time, particularly

535

: 00:34:10

today, is what are all the different ways

that I can spend this dollar and what

536

: 00:34:17

are the most effective, or what's most

likely to give me the biggest impact?

537

: 00:34:23

That's the competition.

538

: 00:34:24

That is the Game of Thrones,

particularly right now in budget season.

539

: 00:34:29

budget season right now is

sort of year round now, right?

540

: 00:34:32

Because everything is so pressed.

541

: 00:34:35

So if you are a marketing leader, you

are in competition to show that money

542

: 00:34:42

spent with you is a better return.

543

: 00:34:46

Than if they spend it in

it or HR or whatever, right

544

: 00:34:50

you also need to really understand

this, you marketing by definition,

545

: 00:34:55

is a non-linear multiplier of areas

of business performance that are

546

: 00:35:00

linear, one of which is sales.

547

: 00:35:02

So in simple language, your leverage,

you are bringing huge amounts of

548

: 00:35:09

leverage to sales performance that

sales cannot create for itself, right?

549

: 00:35:15

not a ding on sales, it's just the nature

of the reality of the situation, what

550

: 00:35:20

do I mean by linear and non-Linear, okay.

551

: 00:35:23

It means that if I were to go

to my CRO and double goal for.

552

: 00:35:28

For 2024, the first conversation

that they're gonna want to have

553

: 00:35:32

with me is about essentially

doubling their Salesforce, right?

554

: 00:35:37

Because the relationship between

revenue coming in and the cost of that

555

: 00:35:44

revenue in the form of sales team is

it's known it's a linear function.

556

: 00:35:52

And that's because it's the

collective performance of a

557

: 00:35:55

lot of individual performances.

558

: 00:35:58

Okay?

559

: 00:35:59

So a bell curve your sales team's

performance is gonna be on a bell curve.

560

: 00:36:04

It's just fundamentally linear.

561

: 00:36:06

The whole reason why modern marketing

was created in: 1890

562

: 00:36:11

to bring non-linear leverage to

that whole equation, even in B2C.

563

: 00:36:18

So what does that mean?

564

: 00:36:19

It means that because of the way

marketing is, were to have the

565

: 00:36:24

same conversation with the CMOA,

we're doubling the revenue goal.

566

: 00:36:29

Might have to increase marketing spend

by 25%, maybe 20%, there's already

567

: 00:36:35

a ton of leverage built into it.

568

: 00:36:37

And if you want an easy understanding

of ROI for marketing, it is the

569

: 00:36:43

extent of that multiplier, so

we'll talk about it this way.

570

: 00:36:47

You are basically saying that

marketing's mission is to help

571

: 00:36:52

sales more product to more people.

572

: 00:36:56

That's revenue faster.

573

: 00:36:58

That's cash flow from

revenue more profitably.

574

: 00:37:02

That's margin impact than

sales could do by itself.

575

: 00:37:07

That's the key phrase.

576

: 00:37:09

The extent to which that is

true is the ROI to the business.

577

: 00:37:13

So does that actually

look like in real life?

578

: 00:37:16

Well, and a lot of really,

really great B2B marketing go-to

579

: 00:37:22

market kinds of operations.

580

: 00:37:24

ratio is somewhere between 10 and 20 x.

581

: 00:37:28

So that means that if you take marketing

away entirely, which I think would be a.

582

: 00:37:34

Something that not even the most draconian

CFO would ever contemplate, but let's

583

: 00:37:39

just say it is the ultimate AB test.

584

: 00:37:42

So we're gonna completely

shut down marketing.

585

: 00:37:44

You're gonna see a massive fall off.

586

: 00:37:46

It may take a year, but you're gonna

see a massive fall off in sales

587

: 00:37:50

productivity that is going to reveal

the extent of the marketing multiplier.

588

: 00:37:54

It's like literally guarantee able because

sales create the leverage for itself.

589

: 00:37:59

It's just not the way it works.

590

: 00:38:01

Justin Norris: If a company is

getting started with this solution

591

: 00:38:04

what would the process be like?

592

: 00:38:06

What data sources would we need to bring?

593

: 00:38:08

How much time does it take to build

the model, that sort of thing.

594

: 00:38:12

Mark Stouse: the very first step is, and

this is part of the onboarding for us,

595

: 00:38:16

is that we sit down with the customer

and we say, look, what are your top

596

: 00:38:21

questions that you most want to know

The answer to this is a mixed audience,

597

: 00:38:26

usually of marketers and business leaders.

598

: 00:38:29

And doesn't matter what your job is.

599

: 00:38:32

Nobody has any problem rattling

off their list of questions.

600

: 00:38:36

Those questions then generate what's

called a model framework, could easily

601

: 00:38:41

analogize that to a recipe card., So this

is gonna be a punch list of data types

602

: 00:38:47

that you're going to need to be able to

supply to the model to compute the answer.

603

: 00:38:55

then it's gonna outline the model

itself, algorithmically speaking.

604

: 00:38:59

When you actually make the dish

from that recipe, that is the model.

605

: 00:39:04

Usually the way this actually

works is it's very fast.

606

: 00:39:08

It's usually a matter of

weeks, like less than a month,

607

: 00:39:12

that kind of timeframe where.

608

: 00:39:15

The analyst and the business user.

609

: 00:39:18

Could be a marketer, could be

finance guy, could be whatever.

610

: 00:39:22

Are collaborating in the tool

on a minimum viable model.

611

: 00:39:26

And at some point business user

says, that answers my question.

612

: 00:39:32

We need to put this model in production.

613

: 00:39:35

You hit the big red button,

it goes into production.

614

: 00:39:38

After that, it's pretty autonomous.

615

: 00:39:42

There's kind of some DevOps type work,

you you know, you have to maintain the

616

: 00:39:46

model and all that kind of stuff, right?

617

: 00:39:48

But for the most part, it is

doing its thing on an automated

618

: 00:39:52

basis and you're getting whatever

cadence is right for your business.

619

: 00:39:59

Daily, weekly, monthly, you're

getting an update on demand.

620

: 00:40:04

Justin Norris: the.

621

: 00:40:04

. Analyst that you just described, is

that someone that you're supplying

622

: 00:40:07

from your team or is this kind

of a bring your own process?

623

: 00:40:10

Mark Stouse: we We have a large partner

ecosystem that, we can recommend from.

624

: 00:40:16

We also occasionally can do it ourselves

on a managed service basis that is

625

: 00:40:23

actually increasingly popular as

teams get thinned out and they know

626

: 00:40:28

that they need this capability, but

they don't have the bandwidth or the

627

: 00:40:32

expertise to manage it internally.

628

: 00:40:36

And so that's actually extremely popular

these days and very cost effective.

629

: 00:40:43

the key thing is that there's lot of.

630

: 00:40:46

You could easily spend 2020 5K

upfront in time, not in licenses.

631

: 00:40:53

Okay.

632

: 00:40:53

But in time to get everything

set up but once you've done that,

633

: 00:40:58

right, again, the models, unless

you need more models, right?

634

: 00:41:02

The models are doing their thing, right?

635

: 00:41:04

You don't have ongoing major, investment,

know, every month gotta redo the

636

: 00:41:10

whole thing from the ground up.

637

: 00:41:12

That's, exactly the kind of thing

that proof was built to eliminate,

638

: 00:41:17

Justin Norris: if you add a

new, you add a new channel, do

639

: 00:41:19

you have to update the model?

640

: 00:41:20

Or the model can be flexible

enough to just say like, oh, you're

641

: 00:41:22

doing LinkedIn advertising now.

642

: 00:41:25

That's fine.

643

: 00:41:26

We can just incorporate that as we go.

644

: 00:41:29

Mark Stouse: you could

do it either way, right?

645

: 00:41:30

I think be the best practice is that

you clone the model that you have and

646

: 00:41:36

then you add the new data streams to it

that you're maintaining the integrity

647

: 00:41:43

of the original model for comparison.

648

: 00:41:46

And yet you are updating

it that way, right?

649

: 00:41:48

And then at some point you're gonna

dispense with the oldest version

650

: 00:41:53

of the model altogether, So it's

not like you just see a massive

651

: 00:41:58

proliferation of models across time.

652

: 00:42:01

But you do need to do this in

an orderly way so that everybody

653

: 00:42:04

doesn't lose reference points.

654

: 00:42:07

Justin Norris: you mentioned

competitors and I've seen competitors

655

: 00:42:09

popping up in the market, even . A

multi-touch attribution vendor

656

: 00:42:13

that's adding MMM to their mix.

657

: 00:42:16

What's your outlook on how easily other

vendors could recreate these capabilities?

658

: 00:42:20

Do you feel that you have a fairly strong

competitive moat around the offering

659

: 00:42:23

you've built, or will other people be

able to jump into this environment and

660

: 00:42:27

offer similar things fairly easily?

661

: 00:42:29

Mark Stouse: So it's really important

to say this, the competitive advantage

662

: 00:42:34

that anybody has is not in the math.

663

: 00:42:36

So if people propose that they have

some kind of Super Cal Flagal algorithm,

664

: 00:42:43

you need to be very careful about that.

665

: 00:42:46

' cause the odds also are that it's

not transparent, it's a black box.

666

: 00:42:50

And also.

667

: 00:42:52

If it is transparent, you probably

don't have the ability to evaluate it.

668

: 00:42:56

In our particular case, the IP that really

matters is in how we automate it, how we

669

: 00:43:04

scale it, how we make it consumable and

approachable and understandable and how

670

: 00:43:10

we do it at a particular price point.

671

: 00:43:12

And I think that this is the other

thing that is highly relevant today,

672

: 00:43:17

not just in this area, but across

SaaS, is that prices are coming down.

673

: 00:43:22

Uh, We're gonna see a fundamental

change in SaaS pricing.

674

: 00:43:27

The days of, you know, every year

having a pricing increase are over.

675

: 00:43:32

It's just done.

676

: 00:43:34

We are very well fixed

competitively speaking.

677

: 00:43:38

Do have a, I think, a really solid moat.

678

: 00:43:40

So there are a lot of competitors today

that have some great products, but these

679

: 00:43:46

are products that were conceived of

and written by and for data scientists.

680

: 00:43:53

and, And a lot of 'em

are gorgeous by the way.

681

: 00:43:55

Like, graphically, they're gorgeous, but

if you expose them to a normal business

682

: 00:44:01

user, a marketer, sales leader, whatever

gonna stare at that screen and go, I have

683

: 00:44:08

not a clue in the world what that means.

684

: 00:44:10

Like how do I make a better

decision based on that?

685

: 00:44:14

You've just now thrown a

lot of friction into it.

686

: 00:44:19

It's taking more time.

687

: 00:44:20

You've gotta spend a lot of time

translating the data science outputs

688

: 00:44:27

into something that is usable.

689

: 00:44:30

And we don't do that, we built it with

the end user in mind violating any.

690

: 00:44:38

Data science principles.

691

: 00:44:39

That's our big advantage.

692

: 00:44:40

Think one of the biggest things I can

say about this, just to sum up, is

693

: 00:44:45

that there's a reason why large B2C

marketing teams, CPG uh, hotel and

694

: 00:44:53

hospitality, retail, whatever, right?

695

: 00:44:56

They control all four pss of marketing and

they have, at worst, a defacto authority

696

: 00:45:05

over and responsibility over the p and l.

697

: 00:45:08

Of their product with their marketing.

698

: 00:45:10

There's a reason for that, they are

using econometrics slash Go-to market

699

: 00:45:16

mix modeling slash MMM, using that to

optimize and to understand causality

700

: 00:45:22

and to optimize based on that causality.

701

: 00:45:25

And they're also investing a ton of

money in market research, which a lot

702

: 00:45:29

of that ends up in the models, right?

703

: 00:45:32

So if B2B marketers want to have the

same attributes their B2C brothers and

704

: 00:45:40

sisters, gonna have to do what B2C does.

705

: 00:45:45

This is one of those situations where it's

like gravity, You can disagree all you

706

: 00:45:51

want to with gravity, and if you throw

yourself off a building, it's not gonna

707

: 00:45:55

end well, the same thing is true here.

708

: 00:45:57

This is a mathematical principle at work.

709

: 00:46:00

It's mathematical law of gravity

with quotation marks around it.

710

: 00:46:05

And so everything that I've

said today, I've, and I really

711

: 00:46:08

try really super hard to do.

712

: 00:46:11

This is not my opinion.

713

: 00:46:13

I'm just representing a level of fact

that people can either accept or reject,

714

: 00:46:19

but it doesn't change the truth of it.

715

: 00:46:22

Justin Norris: We will include a link to

your website so people can uh, check it

716

: 00:46:26

out, learn more, look at your resources.

717

: 00:46:29

And I'm excited to see where this goes.

718

: 00:46:31

Mark, thank you so much

for chatting with me today.

719

: 00:46:34

Mark Stouse: You're welcome.

720

: 00:46:35

Thank you.