Implementation, evaluation and maintenance

Going live with your toolkit is just the start. Now you need to take steps to embed and spread use of the toolkit to ensure that it delivers impact in practice. Key implementation and spread methods include:

Communication 

Mindtools provides a useful learning resource to help you to put together a communication plan

This covers the key elements of:

• Defining your objectives
• Identifying your audience
• Articulating and formatting the messages that will engage each audience.
• Choosing the best communication channels.

The Right Decision Service guide to implementation and spread 

- suggests setting up an Implementation Oversight Group to guide, monitor and support the implementation, spread and evaluation process. 

This guide highlights:

1) Key principles including:

• Shared vision and purpose 
• Leadership at all levels, from the most senior influencers to practitioners at ground -level who will actually use the DSS tools.
• Building internal and external motivation 

2) A suite of implementation and spread tools and techniques, including:

• Improvement methods such as Plan-Do-Study-Act
• Knowledge mobilisation methods such as communities of practice, educational outreach and peer assist meetings.
• Behavioural change techniques including audit and feedback, use of opinion leaders, training and education.
• Managing barriers and facilitators to implementation

Key elements of maintaining your toolkit include:

• the content management plan you defined in your response to the Part 2 toolkit request form
• your response to the post-market surveillance quality assurance criterion - i.e. the processes and responsibilities in place for gathering and responding to user feedback.

You should also check the Content management lifecycle standard operating procedure to make sure you have appropriate processes in place for:

• Change control
• Content withdrawal and archiving

Other key activities to plan for include:

• Responding to reports of broken links - the RDS team will send you monthly reports.
• Testing when there is a new release of the whole Right Decision platform.
• Emergency contact for urgent changes and fixes.

You may wish to establish an editorial group to provide governance and oversight for the ongoing maintenance of your toolkit.

If the Right Decision Service team has built the initial version of your toolkit, it is advisable to organise basic content management training for at least two members of the requesting team, so that they can make update and amendments in a timely way.  The RDS offers group tutorials every month. If the scheduled times do not suit, you can arrange a 1-1 training session by contacting his.decisionsupport@nhs.scot .

Value and impact framework

The Right Decision Service provides a value and impact framework based on an outcomes chain approach. This framework will support you to assess the impact of your toolkit on practitioner knowledge, skills and behaviours; improvement in practice and improvement in outcomes for people, communities and the healthcare system. The framework is summarised in the image below.

Outcomes chain approach

An outcomes chain maps backwards from the long-term outcomes we want to achieve to identify the essential preconditions to achieve those long term goals. It explains the process of change by outlining causal linkages between shorter-term, intermediate, and longer-term outcomes.

Achieving improvement in practice and outcomes (outcome level 4) depends on the outcomes at the earlier levels in the chain.  Achieving uptake and positive response to the toolkit (outcome level 1) is the first stage in impact. It can lead to change in knowledge, skills and confidence (outcome level 2) which then inform change in practice (outcome level 3).  Over time, change in practice will lead to better outcomes for patients and for the health and care system (e.g. in the form of cost savings (outcome level 4.)

Attribution and contribution

For the shorter term outcomes it is relatively straightforward to show a relationship between the outcomes and our services. For example, Reach and reaction are based on data specifically about usage and feedback for RDS products. At these lower outcome levels it is also easier for us to influence the outcomes. For example, we can adapt an RDS tool in response to user feedback to improve uptake and user satisfaction.

As we progress along the outcomes chain, achieving the desired outcomes becomes increasingly dependent on implementation support from our customers and partners in NHS Boards and HSCPs. If there is evidence that practice or outcomes have changed, and we can show that there are high levels of uptake and good feedback for our products at the lower levels of the outcomes chain, then we have a reasoned argument that our products have contributed to the improved outcomes, even if they are not the sole cause.

The Evaluation section of the Right Decision Service implementation guide provides fuller information on:

• Defining the theory of change that underpins the outcomes chain.
• Constructing a logic model to support implementation of the outcomes chain.
• Defining measurable indicators that will provide evidence of impact when it is delivered and support you to identify barriers and make improvements to achieve greater impact when required.

Mindtools learning resource to help you to put together a communication plan

The Right Decision Service guide to implementation and spread  highlights:

• Key principles of implementation and spread
• Improvement methods such as Plan-Do-Study-Act
• Knowledge mobilisation methods such as communities of practice, educational outreach and peer assist meetings.
• Behavioural change techniques including audit and feedback, use of opinion leaders, training and education. 
• Approaches to Managing barriers and facilitators to implementation

Other key elements of maintenance include:

• The content management plan you defined in your response to the Part 2 toolkit request form
• Your response to the post-market surveillance quality assurance criterion - i.e. the processes and responsibilities in place for gathering and responding to user feedback.

You should also check the requirements within the Content management lifecycle standard operating procedure

For which levels in the outcomes chain are you most readily able to gather impact data? 

What do these measures tell you about gaps and opportunities for improvement in the design of your toolkit or the way you implement it to maximise impact?

You may have a colleague locally who can discuss these issues with you.  If you would like to be put in touch with an RDS toolkit manager or editor to share learning and discuss the challenges of impact evaluation, please contact his.decisonsupport@nhs.scot.