Traditional Data ROI and AI ROI

"To know how to dissemble is the knowledge of kings." - Cardinal Richelieu

I know that is a strange quote but I think it is sublime summarisation of a epiphany moment I had.  I am sure all know what ROI Lift in profits minus costs divided by investment equals return on investment (ROI).

Though I had really not thought about how to disassemble the work I had been doing into it's business case. 

Developing new tools is a costly proposition, the type of data and coding projects I most like to do are often out of scope because the assumption that they are expensive.

The other problem that I can see that such data projects are often based on hypotheses that the business has about itself and then puts money into investigating this hypothesis and it is hard to see how the business develops that self awareness if it does not already have it.

I went looking for how does the business use data to find where it needs data.

This is a bit of a blog of my musings on how do I assess the impact of my work. I just thought it was interesting.

A function that enables 

Data's value is often poorly expressed to the company it works mainly as business team that provides business intelligence to other internal customers.

• Data Analytics 
• Data Observability 
• Data Quality 
• Data Warehousing 
• Data Science 
• New Customer: AI

   

   

Each of those customers are project based. When their project finishes the value is accrued to the project.

That was probably the right way of doing that except for the new customer AI.

I am not saying the others do not have value but the possibility of new services, automation makes it particularly of interest to CEOs. Also being a product of data I want to make the argument businesses may want to start trying to estimate their own data's value.

Data often has a range of internal customer who are more sporadic. Problem managers, incidient managers, risk managers all might need data here and there. It is often lots of one off pieces of work.

There is often no Traditional Data ROI being measured

Most important word there is measured... Data has alot of ROI. But I contend we are poorly aware of it.

In fact some members of the data team will read this and say well how do you measure the value of?

• A customer churn dashboard
• A dataset that supports ad hoc queries of engagement behaviours
• Increased data adoption

You see my point at no point has that actually caused a increase in sales or otherwise done something that might be measured by the business.

So each below is a argument I found on how you could measure the impact of data in the business.

Below I want to argue the main sources we could start to measure data's ROI.

• Polling users and customers
• looking where data is mentioned in contracts
• Estimating if data would affect customer retention
• Using incidient, and problem data.
• Asking the board to put a price on what they do or don't know.
• Raiding the Agile method.

Measure the value of data with data

Well the first method is you poll your users and ask them to put a figure in terms of time and or savings. Just ask them what does data mean to them MIT did.

Services like Facebook have been estimated to be worth $40-$50 a month per users in terms of services that the user got for free from them.

Facebook is free, but should it count toward GDP anyway? | MIT News | Massachusetts Institute of Technology 

"To conduct the study, the researchers used three large-scale online surveys in which consumers were asked to put a price tag on the free online services they consume. In many cases, respondents were asked whether they would prefer to keep using a free online good, or to name a price that would compensate for losing access to that product. All told, the surveys drew about 65,000 responses."

A striking line in the article for me was...

"The study also revealed the huge value that consumers place on certain categories of online goods. For instance, people valued search engines at an average of $17,530 per year, and email at $8,414. Collis suggests those numbers may appear so high because many people use search engines and email both at work and in leisure time, and use both factors to assess the overall value."

I started to think businesses should start using these same techniques canvassing managers and customers to get a sense of the value of data to the business.

Customer Facing Lift

There are places in the business where data is the product. This could be that certain data is expressly included as a SLA or KPI. This data is contractual and it's easy to then become aware and put a price tag on it as not having that data has the same value as stipulated in the contract.

This is common enough but contracts often develop, new KPIs are developed but no new contract is signed. This means that data is often no longer linked to the bottom line.

A valid approach might be then where non contractual to estimate the data impact on retention. If losing access to a dashboard would make a customer 1% more likely to leave surely it deserves to be thought about as 1% of that customers value.

The value is almost certainly marginal on a individual dashboard or service so estimation of its impact on retention could allow estimation of the value of work versus its income.

A proxy could be developed in the business of using tickets and their priorities. Rapid raising of P1s might quietly indicate that such a data product is a core feature.

Another proxy might be management based.The people who most talk to the customers could be made proxies of their data needs.

Operational Lift

Data has a operational benefit as well some things are faster and easier when you have the right information and alerts.

Incidient data showing who called and what happened after a data asset outage could be used.

This does assume that such data assets are moment to moment critical though and I think most reports are used weekly or there about.

I feel that it sounds a good idea until you realise most staff can delay using a report for a day or two but lose there mind if it is the last day of the month.

Possibke caveats might be to look for the worst day or event linked to a asset.

Strategic Lift

Board level managers to look back and see if there is any major decisions they would have or not made thanks to data in the company.

This excercise can also look forward expressing what information if had would allow better decision making.

This is usually a great starting point for reflection on starting a series of data projects because being aware of the ask is the first step in finding the data.

The challenge to the strategists though is not to say I'd like to know X I'd pay $200 but to say if I know X I could make us $200 with 50% risk.

The first is more subjective the second is open to challenge and hopefully creates more conversation around the table.

Where could this data be collected

I think there's two approaches to this one is with the business people rolling it into project approaches.

Project level

1. Business Case Creation: This stage involves creating a compelling business case for business or analytics leaders to prioritize use cases. It's also crucial for securing funding from CFOs for execution. Therefore the accountancy function of the business might collect this data.
2.  At Completion: Once the project reaches completion, there's better visibility into its impact. At this stage, it's important to establish ROI at the level of the previously selected KPIs and metrics. Setting targets based on this newfound understanding helps in assessing the project's success.
3. Table owners: someone should own all this data and sponsor the function they might assign it a value though this probably would only reflect their preferences and not actual value. You could always turn it off and see if they notice and that would tell you it's value but for obvious reasons that seems like a bad idea. 

Doing all this should help the team build up knowledge about real term value of their data to the business.

Pleasing or teasing

I have heard a joke that all data functions are either pleasing or teasing.

They either say yes too much and try to please everyone or tease by focusing only on there project work and therefore say no to everyone. 

I think it misses a third option where the data function is more proactive about its work load and I think it's because we do not engage in the above activities.

I think it's striking that data teams often have so little data on what they sell themselves do. The tendency to manage by project and hand team members onto those projects might play a part in this. I have yet to see a data team managed by data outside a few agile tickets.

The agile method itself has stand ups, rituals, and I guess would be a starting place to get that data though would need to have the project using that methodology and already have the tools baked in.

If you do that might be a treasure trove of information.

AI and Automation ROI

When I started this article my intent was to expressly think about how to assess AI value. The problem was I found a striking lack of discussion on ROI, project management or analysis of alternatives.

The thing that occurs to me is that really the value of developers and data has always been badly expressed to the business as according to the above it has barely been tracked if tracked at all and if at all has only been considered as a project cost and not a maintenance cost.

The non recurring nature makes it easily forgotten by the business.

We are used to thinking about the problem as one of master data management between silos of users.

The increased interest in AI might allow the business to leverage its data more to even build products and services out of it.

In AI data is the source of the product (the AI) and the tool interacting with it. It could also fill its own source with garbage synthetic data quite quickly.

That data produced by the AI would have zero worth as it's not a input from the customer.

The part that most concerned me is that without having a price attached to certain data sets how many businesses would do so only 

There appears to be room to express to the business again the value that certain datasets have.

Conclusion

Data is widely reported to be more valuable than oil. But I have yet to seem a company where it's value was even semi regularly appraised.

A criticism I have of the current AI revolution is that it is spoken of as if X gen AI being Y parameters and within Z range of performance is impressive and not ROI.

I also think there is a danger with AI that businesses where the databases store static data will adopt AI not realising it is becoming a more dynamic stack of interacting technologies. Especially where the discourse is the elimination of developers and will be unprepared.

I fear a range of data errors will be covered up and "fixed" with AI but leaving the data silos that cause the errors to occur intact.