Our September newsletter included updates on the Star Online system. We also introduced two new Outcomes Stars for mental health, the My Mind Star and the new and improved edition of the Recovery Star as well as updated research for the new edition of the Recovery Star™
Find out more
My Mind Star™ is an Outcomes Star for building and tracking well-being and resilience in young people and is for young people’s mental health and well-beining
The new and improved edition of the Recovery Star™: the Outcomes Star for mental health and well-being. This Star has been designed to support and measure progress towards recovery for adults experiencing mental health issues and contains changes to make the Star more appropriate, accessible and effective.
In this blog Joy MacKeith, one of Triangle’s founding Directors, explores the interplay of internal and external factors, particularly in the context of austerity in the UK, and sets out Triangle’s approach to the Outcomes Star.
The context of austerity In the 12 years since the first version of the Outcomes Star was published, the political climate in the UK has changed dramatically; funding for services to support vulnerable people has been cut and both employment and housing are increasingly insecure.
The Outcomes Star is a suite of tools which are designed to help service users and service providers work in a constructive partnership toward achieving greater well-being and self-determination. They are designed to be used in the context of an adequately funded service delivered by well trained staff. They are also rooted in the assumption that decent housing and employment are available.
Increasingly in the UK these foundations are not in place and that has understandably led to anger on the part of some service users and service providers. Whilst we believe that the Outcome Star tools continue to have an important role to play in helping people to deal with the challenges they face we also recognise that personal change on its own is not enough and that addressing structural issues of poverty, poor housing, insecure employment and rising inequality are essential to creating a society in which everyone can thrive and contribute.
The agency of the individual In a climate of cuts and reduced services, some may be sceptical about the benefit of services and tools that focus on the agency of the individual. At worse it could feel like people are being asked to ‘pull themselves up by the bootstraps’ or even blamed for their difficulties without adequate recognition of the very real challenges that they face and the sense of despair that can build when the odds are stacked against you (Johnson and Pleace (2016), Friedli and Stearn(20 15)). So how do skills, habits and attitudes – the main focus of the Star, interact with opportunity and life situation? I think the Cycle of Action model presented by NESTA and OSCA in their report ‘Good help: bad help” helps to answer that question.
Their model draws on the ‘COM-B model’ developed by Susan Michie, Director of the Centre for Behaviour Change, University College London. It shows these internal characteristics in a dynamic interaction with a person’s life circumstances so that not only do difficult circumstances decrease confidence, purpose and ability to act, but also these internal states can also impact on outer circumstances.
The implication is that those disadvantaged by difficult life circumstances such as physical or mental health issues, poverty, discrimination or homelessness are likely to also experience less confidence and ability to act and that ‘good help’ which builds these aspects is likely to positively impact on life circumstances. The report is at pains to stress that practical help is also necessary because the external barriers are very real. But the right kind of help, that which builds confidence, sense of purpose and ability to take action, can create a positive, reinforcing virtuous cycle of change. Whilst bad help does the reverse.
The Outcomes Stars have always focused on the agency of the service user. For many of the original Stars, including those for use in the homelessness, mental health and family sectors, the model of change revolves around a shift from an external locus of control (“things happen to me and there’s nothing I can do”) to an internal locus of control (“I want things to be different and there are things I can do to make that happen”).
However, as time has gone on, two things have happened. The first is that we have developed versions of the Star for service user groups who have less control over their circumstances, such as children or those with profound learning disabilities. The second is that the service delivery climate has become more challenging and therefore the external barriers faced by many service users have increased.
For these reasons we increasingly recognise the importance of acknowledging and recording the external barriers as well as supporting the motivation and capability of the service user to overcome barriers. We do this in a number of ways including our guidance for workers on how to use the Star and in the introductions to the User Guides which are used directly with service users . The version of the Star that recognises the importance of external factors the most comprehensively within the tool scales themselves is My Star – the Outcomes Star for children in which some of the scales measure the child’s progress towards resilience and some of the scales measure the extent to which those caring for the child are providing them with what they need to thrive.
Although our intention is to keep the focus of the Stars on supporting agency and ability to act, new editions of existing Stars and new versions are developed with a heightened awareness of the importance of acknowledging the external barriers.
As the NESTA report (page 20) states:
“‘Good help’ is not a substitute for addressing in-work poverty, structural inequalities or discrimination, but it has an important role to play in supporting people to manage the elements that are within their control.”
Our aim in that the Outcomes Star suite of tools enable good help, support people to manage the elements of life that are in their control and help service providers to point to the barriers that are getting in the way so that commissioners and policy makers can play their part in making change possible.
Friedli, L. and Stearn, R (2015) “Positive affect as coercive strategy: conditionality, activation and the role of psychology in UK government workfare programmes’ Critical Medical Humanities. This article does not mention the Outcomes Star but the authors are linked to Recovery in the Bin who have published the ‘Unrecovery Star’ ((https://recoveryinthebin.org/unrecovery-star-2/)
Johnson, G. and Pleace, N. (2016) “How Do We Measure Success in Homelessness Services?: Critically Assessing the Rise of the Homelessness Outcomes Star” European Journal of Homelessness. The focus of this article is a critique of the Homelessness Outcomes Star
Wilson, R. and Cornwell, C. and Flanagan E. and Nielsen, N. and Khan, H. (2018) “Good and bad help: how confidence and purpose transform lives” NESTA and OSCA
The Outcomes Star has been tested psychometrically. A new set of psychometric factsheets demonstrate the validity of the Outcomes Star, and reveal how the Star can produce informative and valuable outcomes data for commissioners, funders and organisations.
Psychometric testing tells us how confident we can be in the data produced by a measurement tool including whether it measures what it claims to measure and produces consistent scores.
Triangle has published a set of factsheets to demonstrate the psychometric properties of every version of the Star. We are also in the process of having an article validating the Family Star Plus published in a peer reviewed journal. Dr Anna Good has produced a psychometric factsheet for each of the Outcomes Stars, providing the findings from a number of these tests. She explains a bit more about the process and importance of the ensuring the Stars are tested psychometrically.
“At its essence, validity means that the information yielded by a test is appropriate, meaningful, and useful for decision making” (Osterlind, 2010, p. 89).
Psychometric validation has been used in some form for over a hundred years. It involves tests of validity (usefulness and meaningfulness) and reliability (consistency), for example:
expert opinion about the content of the measure
clustering of ‘items’ or questions into underlying constructs
consistency across the readings produced by each item
consistency across ‘raters’ using a tool
sensitivity to detect change over time
correlation with, or predicts of, other relevant outcomes
Why is it important to test the Star psychometrically? What are the benefits of testing the Outcomes Star? What’s the background to the research? Triangle recognises the importance of having ‘evidence and theory support the interpretations of test scores’ (AERA, APA & NCME, 1999, p.9), both because we are committed to creating scientifically sound and useful tools and because policy advisors, commissioners and managers require validated outcomes measures and want assurance of a rigorous process of development and testing.
The validation process is an important part of the development of new versions of the Star – we need to know that the outcome areas hang together coherently, whether any outcome areas are unnecessary because of overlap with other areas or have readings that cluster at one end of the Journey of Change.
Once there is sufficient data, we also conduct more extensive psychometric testing using data routinely collected using the published version of the Star. This is beneficial for demonstrating that the Star is responsive to change and that Star readings relate to other outcome measures, which is important both within Triangle and for evidencing the value of our tools externally.
What was involved in producing the psychometric factsheets? The initial validation work for new Stars is conducted using data from collaborators working with Triangle during the Star development and piloting process. It involves collecting Star readings and asking service users and keyworkers to complete questionnaires about the acceptability and how well the Star captures services users’ situations and needs.
The further testing of the published version uses a larger sample size of routinely collected Star data and assesses the sensitivity of the Star in detecting change occurring during engagement with services. Whenever possible, we collaborate with organisations to assess the relationship between Star readings and validated measures or ‘hard outcome measures’ such as school attendance.
We have also been working to assess consistency in worker’s understanding of the scales using a case study method. This method is described fully in an article published in the Journal of Housing Care and Support (MacKeith, 2014), but essentially involves working with organisations using the Star to develop an anonymised case study or ‘service user profile’, and comparing the readings assigned by trained workers with those agreed by a panel of Star experts. The findings tell us how consistent and accurate workers in applying the Star scales when given the same information.
Conclusion: An evidence-based tool The Outcomes Star is an evidence-based tool. The development of new Stars follows a standardized and systematic process of evidence gathering through literature reviews, focus groups, refinement, initial psychometric analyses and full psychometric testing using routinely collected data.
Psychometric validation is useful in the development of new Stars and to provide evidence that the Outcome Star can produce data that meaningfully reflects the construct it is designed to measure.
Organisations can use Triangle’s psychometric factsheets alongside peer reviewed articles to demonstrate the validity of the Outcomes Star to funders and commissioners, and to have confidence that provided it is implemented well, the Star can produce informative and useful data.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing (4th ed.). Washington, DC: American Educational Research Association.
Mackeith, J. (2014). Assessing the reliability of the Outcomes Star in research and practice. Housing, Care and Support, 17(4), 188-197.
Osterlind, S. J. (2010). Modern measurement: Theory, principles, and applications of mental appraisal (2nd ed.). Boston, MA: Pearson Education.
Dr Anna Good: Dr Anna Good is a Research Analyst at Triangle: a large part of her role involves testing the psychometric properties of the Star, conducting research and supporting organisations to make the best use of Star data. After completing an MSc and a PhD in Psychology with specialisms in behaviour change interventions and psychological research methods, Anna spent a number of years as a post-doctoral researcher, including two years as principal investigator on a prestigious grant examining health behaviour change.
Triangle is looking for organisations to collaborate with on research into versions of the Outcomes Star.
We are particularly interested in carrying out psychometric analysis to evidence that the Stars are both valid and reliable measurement tools, but are open to designing the research with you and adding other research questions that are of particular interest to you.
This is an opportunity to be part of original research on the Star, contributing to the growing body of knowledge about the use of different versions of the Star and to debates surrounding outcomes measurement more generally. We will aim to write up and publish the research either as a Triangle document or in a peer reviewed journal.
In return for research collaboration, Triangle can produce a Key Findings Report for your organisation and provide an in-depth analysis of your Star data. Linked to this, collaborators will have access to the new Scale Checker tools and support with Star implementation.
If you are interested in collaborating with Triangle on a research project, please contact usfor an initial discussion, or get in touch directly with our Research Co-ordinator, Emily Lamont – firstname.lastname@example.org.