Research and briefings
There is a growing body of published evidence demonstrating the effectiveness and validity of the Outcomes Star. This research shows that the Star is effective both as a keywork tool and as an outcomes measure, is based on indicators of sustainable impact, and is underpinned by a coherent theory of change and clear values.
The research is summarised in the first of a series of Triangle briefings - The Outcomes Star: Unpacking the evidence.
A bottom-up process of development and testing
The development of the Outcomes Star has been a bottom-up process which has been driven by the needs of service delivery organisations to have practical tools for supporting and measuring change. Each new Star undergoes a very thorough process of testing and revision involving front-line staff, managers and service users. The Organisation Guide for each Star provides a detailed outline of the process and the pilot results. This prize-winning article in Housing Care and Support gives a more general overview of the theoretical underpinnings of the Star and the development process.
Research on the Outcomes Star
In addition to the testing that takes place as part of the Star development process, the following studies have been carried out by Triangle and by external bodies.
This report documents a seminar in which six UK academics, three service providers and one service user came together to share ideas and expertise around the academic studies that have been carried out on the Recovery Star and set the research agenda going forward. The report sets out key conclusions on the validity of the tool and actions to take knowledge forward.
This report documents a seminar in which six organisations using different versions of the Outcomes Star to share their experience of analysing Star data. The report sets out the benefits the organisations had found in using the data and the challenges they faced.
RMIT University Action Research Study of Implementation of Homelessness, Drug and Alcohol and Empowerment Stars
Dr Harris and Dr Andrews of RMIT University, Melbourne Australia carried out an in-depth action research study of the implementation of the Star within St Kilda’s Crisis Support Network. The report identifies 20 benefits to the project of using the Star at organisation, programme, practitioner and client level and concludes that the Star provides a unique innovation in human service delivery.
York Consulting carried out an evaluation of the implementation of the Family Star within Family Action involving the analysis of 3,200 completed Stars, interviews with service users and staff across four locations, interviews with commissioners and Family Action’s management. The study concluded that the Star helped develop service users’ resilience and helped workers reflect on their practice, particularly in terms of improving outcomes and that Family Star data can provide valuable insights into the extent and nature of changes occurring, as well as highlighting areas for further investigation.
St Andrews and Mental Health Provider’s Forum study to investigate the psychometric properties of the Recovery Star
Geoff Dickens, Head of Nursing Research & Research Manager, St Andrew’s Academic Centre, King’s College London, Institute of Psychiatry, Northampton, and Philip Sugarman, CEO & Medical Director, St Andrew’s Academic Centre, King’s College London, Institute of Psychiatry, Northampton, together with Judy Weleminsky, (previously) CEO of the Mental Health Providers Forum, and Yetunde Onifade, Recovery Development Manager at the Mental Health Providers Forum, carried out a study looking at 203 adults who had completed the Recovery Star two or three times (Dickens et al (2012)). They found that the tool had very good internal consistency (Cronbach’s α greater than 0.8), little obvious item redundancy, and most item scores moved in a positive direction over time indicating good responsiveness. They conclude that “Further research on the psychometric properties of Recovery Star is clearly warranted, but this study provides good preliminary evidence for its adoption in recovery-focused mental health services”.
Dr Helen Killaspy and colleagues at the University College London in the UK have recently completed a study to investigate the validity and reliability of the Mental Health Recovery Star. This study was funded by the London Borough of Camden and involved service users and staff from four sites across England.
The study looks at the length of time the tool took to complete, how easy it was to decide on a score, staff and service user views of its usefulness in care planning and as a clinical outcome measure. As well as looking at staff and service user experiences of using the Recovery Star, this study also looked at the test:re-test reliability, inter-rater reliability and the convergent validity of Recovery Star ratings with two other measures.
Read University College London’s study on the Recovery Star.
The methodology and conclusions of this study have been criticized in a number of letters to the British Journal of Psychiatry in which it was published.
Following the 2012 Killaspy et al study, Triangle has developed and piloted an alternative methodology for testing the inter-reliability of the Stars. This involves asking workers to score service user profiles and then comparing their answers with the correct scores (as assigned by a number of expert trainers.) This approach was piloted with 24 workers using the Family Star. The pilot study indicated that the methodology was effective in assessing worker understanding and also provide useful information about the adequacy of the scales on the Star. The pilot indicated that the Family Star has good inter-rater reliability for the five-point Journey of Change, reaching the accepted threshold of 0.8 for the Inter-rater Reliability Coefficient when outlying workers were excluded from the analysis. Triangle has begun a larger study to test the inter-rater reliability of the Family Star amongst a larger group of workers and will be applying the approach to other Stars. For more information see Quality assuring your Star data.
Southside Partnership and the Mental Health Providers Forum (MHPF) undertook a piece of action research in which a Recovery Star Project Coordinator worked with 15 services within Southside to support implementation of the Recovery Star and evaluate the impact of using the Star. The report was published by MHPF in 2009 and found positive impacts on service users, frontline staff and managers.
The Centre for Brain and Mental Health Research (CBMHR) at the University of Newcastle together with the Intermediate Stay Mental Health Unit (ISMHU) and Psychiatric Rehabilitation Service (PRS), Hunter New England Mental Health, Australia and the Schizophrenia Research Institute (SRI), Australia, carried out a multi-component service evaluation project. This focused particularly on questions about the importance of the Recovery Star domains for care planning, and the perceived impact of current treatment practices on these recovery domains. The researchers concluded that “the ten domains extracted from the Mental Health Recovery Star appear to provide a useful basis for examining recovery (at both an individual and a service level)”.
In 2008-9 MHPF carried out a study to look at the effectiveness of the Recovery Star for people from BAME backgrounds, looking at the model’s components through the lens of race and culture. It also examined whether the Recovery Star’s generic models of mental ill health and recovery adequately pick up on and address the role of race and culture on people’s lived experience. The study found that the use of the Recovery Star as part of the key-working session was considered to be a valuable experience by the majority of service users who took part in the pilot. The majority of the areas on the Star were seen to represent sufficient universal themes, giving it cross-cultural relevance. The research showed that improvements could be made by providing further acknowledgement of the cultural practices that shape and impact on the lived experience of mental ill health and as a result of this study a second edition of the Recovery Star was published which addressed the specific issues highlighted in the study.
A study by Burns, MacKeith and Graham (2008) looked at 25 organisations that had been using the homelessness version of the Outcomes Star for two years or more and focused on the interpretative (or practical) aspects of validity. This study found that:
- All 25 organisations reported that key-work had improved as a result of using the Outcomes Star;
- Key-work had become more focused on service user change, more systematic and consistent and covered a wider range of issues than before the Outcomes Star was used;
- Those organisations that analysed the data that they collected found that the data was helpful in assessing the effectiveness of the service and identifying areas for improvement.
In this study the authors used the Outcomes Star for Homelessness with 15 men in a homeless shelter over a 6 week period. Each participant chose two domains on which to focus their change efforts. The study found that participants demonstrated statistically significant change. The authors conclude that the tool is effective in guiding homeless people towards self-reliance and could “change the face of the way that shelters provide and share care with the homeless population.”
A report from the Northern Health and Social Care Trust, Ireland, evaluating a pilot of the Spectrum Star for adults on the autistic spectrum with 13 young adults aged between 14 and 18. Results highlighted that the Spectrum Star is "a very useful tool to use with young people with ASD, given its holistic and visual approach to the assessment" and that it "empowers young people to make decisions with regards to intervention, while still including parents/carers at the action planning stage." The report "recommends the Spectrum Star as a valuable tool to implement into service delivery within the Paediatric ASD Service."
A study of 12 pairs of keyworker and service user experience of using the Life Star, completed in 2015 by Jaina Yasmeen Bharkhda at The Grange Centre for People with Learning Disabilities. The study found that "clients enjoyed the experience of talking about various areas of their life" with the Star, and that the "Journey of Change appeared to resonate with the experience of participants." For workers, the study found that "the Life Star conversations were helpful in understanding their clients" and that "implementing the Life Star at The Grange compliments and accentuates the existing Person-Centred approach." In addition, the study found that "the Life Star's emphasis on collaboration promotes more choice and control for the client, which ultimately leads to empowerment." The study also found, however, a degree of scepticism from both workers and clients, around the purpose of the Star and concerns over data privacy and use - highlighting the importance of clear communication and engagement with staff and service users when implementing the Star.
Developing a common approach to supporting the progression of young people towards employment: Pilot Project using ‘Work Star’, London Borough of Hounslow 2015
The final report of a project evaluating the use of the Work Star with vulnerable 16+ young adults across 6 services including the Connexions team and local, partner organisations. The study found that feedback from partners involved the pilot was “overwhelmingly positive,” with particularly strong endorsements for the way the Work Star created “a consistent approach within and between organisations in how we describe where particular clients are on their journey towards employment” and the opportunities it gives for “line managers to systematically review vase work and engage in outcome-focused discussions during supervisions linked to specific casework.”