How do you go about measuring the effectiveness of your learning and development solutions? And what do you measure?

As learning and development professionals it’s important to look at the impact of the learning solutions we provide our learners instead of just focusing on completion rates, assessment scores and how many times our learners have logged into our learning systems.

So, let’s take a look at some of the common models that could help you measure the impact of your learning solutions.

The Kirkpatrick Model

Have you heard of The Kirkpatrick Model? It’s a widely used four-level training evaluation method first published by Donald Kirkpatrick in 1959 and has since become the industry standard in L&D.

The four levels of The Kirkpatrick Model are:

Reaction

The first level assesses learners’ reactions to the training. This is usually done through surveys and feedback forms.

Learning

Level 2 measures the increase in knowledge and capability experienced by learners and is often measured through tests, interviews, or observation.

Behaviour

The third level evaluates the extent to which learners apply their learning in the workplace and this is done through observation, interviews, and 180- or 360-degree feedback.

Results

And finally, Level 4 assesses the overall impact of the learner’s performance on the business or working environment. This level requires a unit of change in learning to be linked directly to a specific improvement in a key organisational metric.

Although it has been updated, the Kirkpatrick Model is now more than sixty years old. It was designed for a traditional, active, group training model, whereas a large proportion of training in the 21st century happens individually, during ordinary work activities and virtually.

Despite this, the Kirkpatrick Model for training and development evaluation remains one of the best out there, and it’s still widely adopted to validate training endeavours.

The Phillips Model

The Phillips ROI Model is often considered to be the fifth level of the Kirkpatrick Model and focuses on evaluating the return on investment (ROI) of the program.

The ROI calculation is a financial measure that assesses the economic value of the program, by comparing the costs of the program to the benefits that it generates. It is a way to demonstrate the impact of the training or development program in financial terms, and it’s a way to show the organisation if the program is worth the investment.

By adding the Phillips ROI Model as a level 5 to the Kirkpatrick Model, organisations can not only evaluate the effectiveness of the program in terms of reaction, learning, behaviour and results, but also assess the economic value it has generated for the organisation, by comparing the costs of the program with the benefits generated.

This allows the organisation to make more informed decisions about the future of the program, whether to continue, modify or discontinue the program.

The Level 5 evaluation equation looks a little like the following:

ROI % = (£ Benefit of Training – £ Cost of Training) / Cost of Training

The equation can be used to help you develop ROI forecasts and projections and enable your organisation to make more informed training investment decisions.
 

The Kaufman Model

Kaufman’s Five Levels of Evaluation Model, introduced by Roger Kaufman in 1984, is positioned as a more practical and effective approach to evaluation than Kirkpatrick’s model.

It is an extension and development of the Kirkpatrick Model of Evaluation, with similar evaluation processes. The five levels of Kaufman’s model are:

Input
The first step focuses on the measurement of resources. To assess what resources are available and used to support the training and the coaching.

Workshops and research are key during this step to ensure a thorough evaluation of the training resources used.

Process
The second split of the first step, is the evaluation of the overall delivery process.

Survey’s, feedback forms or live observation are some key examples of how to go about assessing this stage.

Acquisition
This step, referred to by Kaufman as a micro-level, measures how much of the learning is absorbed by the learners.

Some things that come to mind assessing this step, is to hold tests for learners to ensure they have “acquired” and remembered what was in the training.

Application
This stage is also referred to as another ‘micro-level’.

In this stage the learners would be observed applying what they learnt to their roles around the workplace – outside of tests.

This can be monitored through live observation around the workplace – an example being, how does a learner deliver customer service after customer service training.

Organisational Payoffs
We then move onto the payoffs for the organisation (the macro-level client) as a whole.

Rather than just evaluate the learner, Kaufman’s model also suggests evaluating the impact the training has had on the organisation as well.

For example, has the training generated an improvement in performance across the board, or has it helped with growth/income across the organisation.

Societal Outcomes
Kaufman’s final step outlines the benefits the training might have on other organisations or society in general.

Does the training transcend out to external business clients and into society? Happier customers, growth in a particular market etc.

Whilst Kaufman’s Five Levels of Evaluation may not be everyone’s go-to for training evaluation (with Kirkpatrick’s taking precedent), the model does offer some additional concepts that can be applied to ensure a thorough evaluation of L&D initiatives.

The CIRO Model

The CIRO Model, developed by Peter Warr, Michael Bird, and Neil Rackham in 1980, offers an alternative to Kirkpatrick and Kaufman’s evaluation models, specifically for management training.

CIRO stands for Context, Input, Reaction and Outcome and like most learning evaluation models it follows a hierarchical structure. This means you would start with Context, then move to Input, and through the others one at a time.

Context
The Context step assesses the operational situation and determines the training needs.

Input
The Input step analyzes the resources and delivery of the training.

Reaction
The Reaction step gathers feedback from learners about the content, approach, and value of the training.

Outcome
The Outcome step evaluates the success of the training in three categories: Immediate Level, Intermediate Level, and Ultimate Level.

While the CIRO Model is a niche model that may not be flexible enough to measure alternative roles and positions, it provides a bespoke approach for evaluating management and leadership training.

It’s key to note that in this model, the first three levels are evaluation steps and the last one provides the results.

You cannot see the results without first evaluating the first three steps.

The Brinkerhoff Model

The Brinkerhoff Model, also known as the Success Case Method, is a recent method of evaluating training introduced by Robert Brinkerhoff in 2003. The model focuses on identifying the best and worst learners who participated in the training to determine what worked and what didn’t. It places a strong emphasis on qualitative analysis rather than surveys or assessment data.

The five steps of the Brinkerhoff Model include

Planning
Determine what programme or course you want to evaluate.

Defining success
Make an impact model for the programme/course and define what success should look like for learners taking part in the programme/course.

Identifying the most and least successful learners
Create a survey/feedback form or any other method to identify the most and least successful learners.

For example, who scored the most/least on a test initiated directly after the training.

Documenting evidence
Conduct interviews off the back of the findings with the most and least successful learners and document why they succeeded/failed or why they enjoyed/didn’t enjoy the training. Bring in supporting evidence.

Concluding and making recommendations
Communicate your findings with any relevant parties involved – HR, L&D teams – and share the lessons learnt. From here you can make any recommendations for improvement or hold group sessions to improve the training.

By drawing on the extremes of best and worst, Brinkerhoff suggests that the model can help to identify what is really happening in the training, and how the training can be improved.

One of the key strengths of the Brinkerhoff Model is that it provides a clear and systematic approach for evaluating development projects and allows improvements to be identified early on, which can help to increase the chances of success.

The Anderson Model

The Anderson Model, also known as Anderson’s Value of Learning Model, is a relatively new training evaluation model designed for evaluating learning at the organisational level. It aims to align an organisation’s learning solutions with its strategic priorities. The model has three stages:

Stage 1
Determine the current alignment of learning against strategic priorities

Stage 2
Assess and evaluate the contribution of learning to the organisation

Stage 3
Establish the most relevant approaches

Anderson’s model focuses on evaluating the effectiveness of an organisation’s learning and development strategy, using measurements such as the effectiveness of the L&Ds function, return on expectation, ROI, and benchmarking.

The model emphasises the importance of metrics and tailoring the approach to the unique needs and values of the organisation.

Unlike other models such as Kirkpatrick and Phillips, Anderson’s model does not focus on specific learning events but on developing a learning and development strategy linked to organisational performance improvement. The Anderson Model and other models such as Kirkpatrick’s can be used together to evaluate learning at both the micro and macro levels.

The Learning-Transfer Evaluation Model (LTEM)

The LTEM, introduced by Dr. Will Thalheimer in 2018, is a model consisting of eight tiers for evaluating the effectiveness of training. The eight tiers are as follows:

Tier 1 – Attendance
The learner signs up/attends a learning experience or training of some sort.

Tier 2 – Activity
The learner engages in the training experience and takes part.

Tier 3 – Learner Perceptions
At Tier 3 you would question the learners about the training they had taken part in to gain insight into how effective it was for them – You could do this through feedback forms, surveys etc.

Tier 4 – Knowledge
Tier four measures the extent to which learners remember what they were taught.

Tier 5 – Decision Making Competence
The learner makes decisions given relevant realistic scenarios – i.e., roleplay or any other form of testing where what was taught can be put into action.

Tier 6 – Task Competence
The learner is tested and performs relevant realistic actions and decision making to determine how competent they are with the new skills they have required.

Tier 7 – Transfer
The seventh tier is to measure when a learner uses what was learned to perform work tasks successfully on a day-to-day basis.

Tier 8 – Effects of Transfer
The last tier measures the effects of ‘Transfer’.

What are the effects of the transfer on the (a) learners, (b) co-worker, family, and friends, (c) organisation, (d) community, (e) society, and (f) the environment.

Thalheimer designed the LTEM with the goal of improving job performance, emphasizing the importance of learners applying their new skills in their work and personal lives. While Kirkpatrick’s Evaluation Model is considered the industry standard for evaluating training, the LTEM has gained increasing recognition since its introduction.

Conclusion

Measuring the effectiveness of learning and development solutions is essential to ensuring that training investments deliver a measurable return. However, choosing the right methodology to measure learning impact is crucial.

Hopefully the models above give you the inspiration and direction to look at the impact of your learning solutions in a new light and help to make it even more of a success.