How to measure the ROI of data analytics in your company

If you’ve read our thoughts on how to build an efficient data team, you already know that there are a few truths we hold dear here at AirOps, mainly... 

• An analytics engineer should be your first data team hire.
• Data leaders should be prepared to address the ROI of their data function and data stack.

Measuring the return on investment (ROI) of a data team is a contentious topic. It’s not always straightforward and a lot has been written about the different frameworks and metrics you can use.

Sometimes, data’s impact can be measured directly, like if you’re selling data as a product. In most cases, however, the data function creates indirect value as it supports marketing, sales, product, and other functions throughout the organization.

This makes it difficult to measure the economic impact of data… which will be a problem if you’re hoping to justify further investments in people and tooling. 

Just because it’s difficult doesn’t mean it’s impossible, though, which is why we sat down with Chris Meier, Director of Data at Bambee to get his take on the topic. Keep reading to get his insights on how to measure the ROI of data analytics, including best practices, which metrics to look at, and more.

‍

Measuring the ROI of a data team and/or data stack is often described as “difficult”. Why do you think that is?

I don't know that it’s difficult, but it is different. When you think about ROI, most people want to tie a dollar value to the work that's being done. 

Many folks think about the ROI of data analytics the same way they think about ROI of sales or marketing: We know how much we're spending on the sales team. We know what we’re spending on marketing. We know how many dollars we're getting back from both functions. It’s pretty straightforward. 

The data team lives in a weird Limboland that’s not dissimilar to finance and operations, where their role isn't necessarily about creating revenue. It’s about reporting on things that are important to the business and driving efficiency. We know there’s value in that work, but there aren’t dollars tied to it. I think that the data team is somewhat similar. Although it’s possible to tie the outcomes of an analysis or the data team's efforts to business value, it's not always super clear. 

For example, I could build a dashboard for somebody and they could review that dashboard and use it to make a decision that’s data-backed versus going with their gut instincts, and let’s say that saves a customer from churning. So my work led to a customer not churning. It’s a very tangible outcome, but drawing the line to get there is more challenging. The attribution of return for the data function is more challenging than it is with many other functions in a business. 

It's not going to be a one-to-one relationship of “data team did X and efficiency did Y.” But we can use a bit of an artistic approach to answer the ROI question. We can talk to stakeholders to understand exactly what value we’ve delivered. 

Best practices for measuring ROI of data analytics

What are some of the most common ways that organizations totally blow it and get upside down on the ROI of their data stack/data function? 

Having done this twice, maybe even three times now, I think there's a really good way to get ROI from a data team and a terrible way to do it.

The approach that I have seen go really poorly is trying to build everything all at once. I think analytics engineers are people who like to build things. That's kind of why we found ourselves nestled into this career. We like the in-between of providing analytics but also building something. It's not really engineering, but it also kind of is. You can create these crazy data stacks that are quite impressive and complex. 

I’ve seen organizations run into trouble when they say, “We need to build a solution that’s going to allow us to perform analytics on every piece of the business.” And then they start mapping everything out. Four or five or six months into the project, they realize they’ve done nothing but plan. Or maybe the team has started to build, but only specific pieces that are so interdependent upon each other that having built one little piece doesn't necessarily allow them to deliver value. 

Instead, data teams should be asking, “What piece of value is the most important to the business right now? What is the highest business priority and how am I serving that?”

Basically, don’t blow it and try to boil the ocean. 

However, you still should have some consideration and some peripheral vision as to what you're going to need in the long term. A simple example of this would be somebody asking you to analyze communications with customers and they're specifically interested in how often we call our customers. So you go and ingest data from your sources and you bring it into the warehouse. Then you transform that data in a way that serves the analysis you're doing at the time. 

I think you can, and should, step back and say, “How do I want to build this thing to serve a larger outcome?” I think there are ways that you can be considerate of the larger picture of what the data team is attempting to serve across the entire organization, but still deliver value specifically in that moment and do it quickly. 

In the example I just gave about communications, I may want to analyze calls right now, but I also know there are other ways we talk to our customers, like text messages, emails, and live chat. So I’m also thinking about how to model the data or how to create a schema that will serve all of those things down the road. 

On the flip side, what are some of the “good” things that organizations can do to make sure they get positive ROI from their data investments? 

Focus on delivering value first. 

There are certain projects that would be high value for the business. That should get some weight in terms of prioritization, focusing on whatever is the highest priority for the business and aligning your projects to whatever is going to drive outcomes that will drive business value. 

In terms of what that actually looks like, it’s typically that you're: 

• unlocking somebody's ability to make a decision;
• driving efficiencies within the business, or; 
• uncovering revenue-generating opportunities. 

If you focus on those three things and you're aligned to the business value, it's typically pretty easy to say what the ROI was. Usually, the business has done a lot of the heavy lifting to determine why something is important and it's often backed by some dollar amount. 

Being able to say, “We are aligning ourselves to this business objective because it’s associated with some dollar amount and without data we’re not going to be able to accomplish this quickly or at all” or, “We think we can squeeze out some more efficiencies based on and analysis” makes it pretty easy to show how the data team provides value.

Metrics to help measure the ROI of data analytics

What are the most important performance measures and metrics you would look at to assess the performance of a data stack as a whole?

Since we don't measure ourselves on revenue and we don’t measure our efficiency gains, the thing that’s most important for my team is turnaround time. 

The data has to be accurate so there's a whole set of metrics you could assess around data quality. But for us, it primarily boils down to how quickly we can turn around information and get somebody unblocked on their decision. If our CEO asks a question and has to wait two weeks to figure out the answer, that sucks. If we can get him an answer within the day and he can make a decision that night, and then tomorrow the product team can start executing on that.

Also, the three areas that I mentioned before: unlocking the ability to make a decision, identifying efficiency opportunities, and identifying revenue opportunities, all have different ways to measure how much you got out of that investment. 

If you're looking at how data unlocks somebody's ability to make a decision, that can be measured through a conversation with a stakeholder. You can ask, “What was the cost of not making that decision, and what did we gain from having made that decision?” 

It's gonna be a little bit tricky to measure but oftentimes a conversation with a stakeholder will reveal some insight. Here's a good example: We were ramping up account executives and our leadership wanted to understand if the quotas set for our newest Account Executives (AEs) were appropriate given the performance of past AEs and other factors. It turns out that quotas do need to be adjusted given a multitude of factors and not just a set percent of our tenured AEs. With that new lens, we saw some AEs that were new and actually doing really well. Because the data team was able to provide information, our  sales leadership was able to make a decision. From that decision, our sales team will become more efficient or these people will produce more revenue. 

There are many situations where you can't tie it back to dollars saved or dollars gained. But looking at what decisions were made and their impacts is a totally fine way to measure ROI for a data team in that scenario. You can also tell the story of what the team did rather than try to say something like, “Here's the number for our team that says we did well.”

Relationship-building and ROI

We love that concept, “The best way to get people to tell the data ROI story is to get other people to tell it.” How should data leaders and their teams go about developing those relationships?

I think there are two sides to this:

• One is deciding that you're collecting feedback from the stakeholders who determine what the value was that you delivered. 
• Or, there’s working with stakeholders and getting them to talk about you. 

Ideally, you work with a business stakeholder, then they go show off what they did accompanied by the data team’s work and say, “By the way, this team helped us out with it.” 

Getting people to talk about the data team when they aren’t around comes from having a good culture within the company. The idea of talking about shared successes is incredibly important for this to happen. 

In the first scenario where you're collecting feedback on your work. I think data teams are most successful when they're closely tied to the business. Which means that rather than being seen as a report writer in a support function, you're seen more as a strategic partner. 

Here’s what that typically looks like: 

• You're having meetings with department heads on a regular cadence.
• You're in tune with everything that they have going on and their new initiatives.
• You're able to say, “Here are ways that we can get more efficient with your team,” or “Here's how you can save more time,” or whatever the goal may be. 

So, you’re shifting away from the “here's the piece of data, use it to make a decision” mindset and instead being very in tune with the business and understanding what's going on and where you can provide value. Not only are you going to be able to see the impact of your work, but you're also going to get direct feedback. 

More often than not, you're also providing a recommendation on what stakeholders should do. After that, you're helping them measure whether or not the thing that you suggested did what you expected. In which case it's very clear whether the data team achieved the goal. 

Either way, you want to be a strategic partner to your peers, company teams, and your stakeholders.

Analytics engineers ❤️

We’re big fans of the analytics engineering skillset. We often recommend that startups, early-stage companies, and other organizations hire an analytics engineer to be the first person on a new data team. What are your thoughts on that approach and what should organizations be aware of when hiring?

Hiring an analytics engineer is probably the right call. They can do the analytics piece and turn around some charts and help make decisions, but they can also set up a data stack to some degree, especially given the landscape of cloud data tooling. It wouldn't be challenging for an analytics engineer to set that all up.

There are a few things to look out for though. If you get somebody who hasn't spent enough time on the analytics side, they often don't understand the business value that you're trying to get out of the data stack. And that becomes critically important in terms of everything we talked about today – ensuring that everything you're doing is aligning to business value. It's not always clear to somebody who's only done the data modeling side, how that data model might translate to business value. It's not always clear what decisions are being made or what the impact of those decisions are. 

It's very easy to get into this mindset where you just want to build everything perfectly and have this lovely data model. You don't realize that whether or not you have a couple of perfect fields doesn't really matter at the end of the day. Knowing when to spend a bunch of time on something versus not is somewhat informed by having a background of working closely with the business and running analyses over and over and over again and working with stakeholders and knowing how they think. 

That’s a valid point about making sure that the analytics engineer is fluent on both sides of the continuum. In your opinion, do you ever see analytics engineers working in a silo, and do you have any suggestions on how to bring them back into the business? Also, any tips on how to avoid that problem to begin with?

I know this sounds really simple, but having the analytics engineer talk with the business on a regular cadence is the way to get that done. The only people that are going to tell you how valuable your work is are the people that are actually using it.

And so if you get somebody who's too deep in the middle of the stack and you need to yank them over to the business side, you want to encourage them to figure out why they’re providing someone with data. They should be able to answer questions like: Why do they need a dashboard? Why do they need a dataset? What decisions are we driving? Those questions are a good way to pull them over and then say, “Okay, we need to find new opportunities to deliver value to these teams.” 

This is more of a coaching mechanism than a team strategy, but I’ve found it to be an effective way to un-silo an analytics engineer.

In a recent blog on building an efficient data team, we used a crawl → walk → run analogy to describe how an analytics engineer can help organizations build a strong data function. We described running as doing the “cool stuff,” like building machine learning (ML) models and integrating artificial intelligence (AI) into your product. What are your thoughts on that? 

When I first started my data career, my perception was that ML was the pinnacle of everything that we do. What I have found is that it is a very rare day that I build those things.

There's a lot of value that comes from what you would call the walk phase. We deliver most of our value in that phase. The value that ML might provide is a fraction of a percent compared to everything else we do currently. 

There are interesting applications, but I guess it depends on what your data team is focused on. If you’re providing a product back to the customer then yeah, ML is a really interesting path to go down and there's probably a lot of opportunity there. 

But, if your data team is mostly focused on business operations, identifying revenue opportunities, and unlocking decisions, then ML probably isn't worth the investment until you are very data mature.

Now that you’ve been working in the data analytics field for a while now, do you view the “walk” phase as inherently cool on its own?

Absolutely.

I think the one thing I've learned about the data role is that there are an incredible number of ways to analyze data and deliver value. If you deliver a dashboard and then help somebody use that and then determine that they are saving a bunch of time or they're making more money as a result of it. That's really cool.

‍