It feels like we’re in the future, doesn’t it? We’re surrounded by incredible technologies, to the point we’ve grown used to them, to the point we may not realize they exist. Are learning and education also part of such groundbreaking future? Will students and teachers experience higher levels of engagement, effortlessly? Will leaders find paths for their communities to thrive, in ways and rates never before seen?
And will we all have Artificial Intelligence to thank for thrusting us forward?
Along the so called hype cycle, many are still riding the AI wave. For an increasing number of people, expectations peaked already. (Where in the curve are you now?)
In this article, we will discuss examples in which an ROI in Learning process can be effectively implemented to help organizations figure out its value, and ultimately figure themselves out along the way. We will also discuss practical ways to incorporate an ROI mindset to things like AI (“Return On AI” or ROAI) in order to understand their actual benefits.
Download the Open LMS ebook “Measuring Impact and ROI: How a Corporate LMS Helps You Get Started”
Put on your financial hat (just for a second, I promise)
We can only know what investments, costs and efforts are worth it; or understand the impact of any software, tool or initiative, however well intentioned, by having an objective, comparable evaluation framework. One we can administer systematically.
So let’s take AI Whatever help AI brings must check two boxes:
- The benefits must outweigh its costs. Makes sense, yes? Just be careful to include things beyond the product itself, but needed to get value out of it. There is onboarding, risk assessments, technical debt, and the internal time spent by the team on getting it to work.
- The net benefits must be higher than any other alternative out there. So we’re back to the important idea of comparability, also known as opportunity cost. It may not be enough that we’re getting more out than we put in, but that the resources we’re putting in to make it work are not better spent elsewhere.
These are two reasonable conditions for successful innovation. Along the way, making sure we are on top of these should help us figure out what makes sense supporting, because we’re benefitting from it.
Otherwise, the reason for introducing a tech like AI is not tied to a common benefit. But how can we really tell?
If only there was a way, more or less objective, to stack up investment ideas against one another, to see which ones are worth our time…
Behold: Return On Investment (ROI). Originally an accounting indicator, over the course of the XX century this operational ratio became a staple of marketing, project management and pretty much any area where efficacious use of resources matters.
In recent years, as more people and groups in society find new value and understanding in the ROI, its scope has only grown larger, and ideas like ROI on Learning start to populate the tool belt of educators and eLearning professionals.
Now, as AI thrust us into another new normal with unprecedented speed, the it makes sense to take a step back and figure out a way to tell, once and for all, if it’s worth it. But before we can move forward with the process, we need to make sure of a couple things:
- We need to adopt a mindset that allows us to evaluate different “things.” By things we can mean “innovations,” but depending on your organization it would be more practical to think in terms of “projects,” “initiatives” or “Products,” which is what we’ll be using from now on.
- We need to identify at least two products we can compare.
- Products must be comparable, both in costs and benefits. Both costs and financial benefits have monetary units, making it easy to calculate ROIs.
- For non-monetary benefits, we need to find a way to compare their efficacy at providing said benefits. Each type of non-monetary benefit becomes another dimension of the ROI in Learning. It is optional (and cumbersome) to try and put a price to them, but there are ways to create quantitative approaches and to evaluate the efficacy in relation to their (monetary) costs.
- Finally, we need to create a way to build a composite index that help us appreciate the different types of benefits across dimensions, for the different products.
- Last but not least, the process should be easily integrated into workflows, especially where innovative teams play a part. And of course, the process should never replace people’s judgement, but empower it.
Standard Return on Investment (ROI)
ROI = (Revenue – Cost) / Cost × 100
Now, onto the market of AI products and AI product-making tools
I’ve always believed that part of the innovator mindset is the desire to become a connoisseur. Someone who is willing to try new, unexpected things, because they know overall those will lead to a better quality of living, and the ability to be able to try new things, thus perpetuating the virtuous cycle.
As it turns out, skill and affinity towards trying out new things are more relevant than ever in the marketplace-centric innovation space we find ourselves in today. Developing innovations in-house, including custom built AI is perfectly plausible, perhaps more than ever. Still, the volume of options for hire is leading to interesting solutions that are likely to work for most cases with little to no extra customization.
But beware. Because the market is not operating at the rigorous levels of efficiency we economists expect it should. There are frictions that keep the real world away from the perfectly transparent flow we were raised with.
Which could be but the start of the issues. Even with perfect information, there are challenges due to the complexity of products and use cases, making the decision process cumbersome and expensive, often to the point of becoming as difficult as in-house product development. It’s no wonder simplicity is such a sought after feature, for both seller and buyer.
By the way, here could lie a promising, unique potential boon for AI: Helping teams and communities find alternatives that increase the quality of living, or in our case the learner experience, by increasing our readiness to reap rewards from new products that prove beneficial. Consumer pressure could force all market competitors to provide increasing level of transparency in their offers, perhaps following the practices of the cloud business. AI can speed up parts of the process of identifying a problem, evaluating data and estimating the benefits and drawbacks of any new venture, constantly. AI could do it faster and more frequently.
AI could become your automated Risk Manager Officer who doubles as your Process Improvement Consultant.
The subtle art of knowing what you care about: Quantifying the unquantifiable through marginal analysis
Dimensional ROI in Learning
Revenue with a non-monetary,
quantitative measure of Benefits
Eg. Number of certified students
ROIi = Benefitsi / Costsi
i is the total of dimensions to consider.
Repeat for as many dimensions as desired
As AI has shown, the role of new innovation in the classroom cannot be analyzed only through financial lenses. A host of issues, ranging from the propensity to cheating and other ethical concerns, the impact on jobs, an even environmental impacts, need to be accounted for.
So, what kind of magical box should we throw our ideas into, so the box returns the best one?
If you thought the answer was ROI… you’d be right!
Dimensional ROIs allows us to compare benefits in quantitative, but not monetary ways. There are a number of reasonable requirements involved:
- The benefit must be measurable and their units must be the same for every Product in the comparison. Call it the “apples to oranges” condition, if you will.
- In most cases, it’s best practice to calculate the benefits over a given period of time. Each product and use case has a period of time in which the amount of benefits is normalized and comparable.
- Unlike the Revenue, which is a net value, the Benefits variable takes into account the drawbacks of the implementation. Recently economists have found ways to estimate environmental costs required to alleviate a given consequence (or negative externality).
Up until now, there has been a subtext living behind the concepts and the process of coming up with a fair and transparent method to identify worthwhile investments and innovations. There is a very wobbly-looking, subjective basis for the rigorous, quantitative analysis we are proposing. In the freedom of every organization to pursue their unique goals, dwells their freedom to define benefits and their importance.
Truth of the matter is, successfully innovative organizations achieve this by taming the wobbliness down to zero. The ventures of an organization with an unwavering mission gain the public’s trust more easily.
Taking it forward: ROI competency paths
We’ve briefly described practical ways to implement ROI and and “ROI mindset” for eLearning processes. We’ve made some emphasis on things that either make coming up with the figure easier, so it feels approachable and relatable. We’ve also highlighted ways in which the ROI process can evolve within your organization, often alongside complexity. But let us be clear: When it comes to financial performance management, simplicity is always best until it stops becoming an adequate reflection of reality.
In this sense, let’s list the “competency paths” for ROI in Learning, as the fields of study, practice and research that would leads to manageable complexity and better outcomes.
- The Advocacy path: Take a page off this article, and let’s continue to think in ways ROI and other financial concepts and ratios seep into our communities and the industry’s everyday conversations. Steps in this path may include experimentation, cost awareness and accountability; and a quantitative-friendly.
- The Market-Making path: It is clear that, for all its widely recognized benefits, the biggest gap towards ROI adoption is the limited number of available tools and user experiences. An ROI in Learning product could take the form of software or consultancy.
- The Dimensionality path: When it comes to quantifying the unquantifiable, and doing it right, there is plenty of work to be done in standardization, unifying a language. There is promise —and peril— in examples such as Ecosystem Services, to flesh out the value of environmental assets, as some hope to provide a better economic context to our current industries.
- The Marginal analysis path: A growing number of “radical” financial analysts seem to borrow a page from day-traders, and evaluate ROI to near real-time frequency, across a larger number of smaller investments. This would not only lead to more efficient ROI-related routines, but to a whole new dimension for the study and research of investment and return dynamics, society-wide.
ROI Scoring
For all i dimensions,
determine their importance or weight
It is recommended to choose the weights
in a scale of 0-100%
Do not choose the highest-scoring blindly; look at the benefits between alternatives as a whole
Back to the here and now — To Sum Up
If we had to pick a different beginning for our story, a likely better one, we’d begin by emphasizing the importance of a clear, unwavering mission for an innovative organization. This would simplify their process of evaluating risk, and make their results more trustworthy. An importante side-message: This also helps explain why, when it comes to true innovation, competition is rarely a concern, as each new innovation addresses a problem hitherto unaddressed in society.
Understanding of the company’s mission should lead to the identification of dimensions that matter, and ways to quantify a benefit for said dimension.
Once solutions have been diagnosed, product evaluation can follow. It is common for RFPs to halt if the number of viable alternatives competing is below a threshold. Viability may include the product’s ability to provide precise data about to generate comparable cost-benefit measurements.
In the future, AI could spark a revolution in the way and speed organizations evaluate and incorporate multi-dimensional benefits available in the market.
With all the ingredients present, the ROI in Learning analysis can proceed. Clarity in the organization’s mission translates to transparency, allowing more and more members of the community to participate and reproduce the process.
At the final moment of evaluation it is perhaps most important to remember than ROI in Learning is a decision-making input, nor a substitute. Here lies the value of organizational leadership, as the captain of the ship who sees the metrics of the wind, but ultimately makes the decision of where to take.
Download the Open LMS ebook “Measuring Impact and ROI: How a Corporate LMS Helps You Get Started”
