As the creators of a suite of measures capturing distance travelled towards ‘hard outcomes’ we are sometimes asked whether there is evidence that Star readings correlate with or predict outcomes such as offending or employment. In some cases, we hear there is resistance to using the Star and instead commissioners, managers or funders are only interested in how many service users have ticked the box of meeting these hard outcomes. This misses out on capturing important achievements, ignores the role of internal change in maintaining concrete achievements and disincentives working with those most in need of support.
This briefing describes some of the evidence we have of the ‘predictive validity’ of the Star – that it does in fact predict outcomes such as school attendance, employment, training and accommodation status. This includes findings reported in two articles recently published in peer-reviewed journals.
In it, we also explain the value of the Outcomes Star in measuring the full journey leading up to and including changes in behaviour or circumstances.
The author of this briefing, Dr Anna Good, draws on her expertise in behaviour change theory to summarise the strong evidence base supporting the importance of the changes assessed by the Star. It is clear from the research literature (and our extensive experience of working with service providers), that early steps on the Star’s ‘Journey of Change’ such as acknowledging problems and accepting help are often essential to subsequent change in hard outcomes. Moreover, change in skills, confidence and beliefs are often key factors in the maintenance of life-changing improvements.
The Outcomes Star is well established as a tool for supporting effective keywork and demonstrating achievements. Here, 'Triangle's Research Analyst, Dr Anna Good, discusses a third benefit, the opportunity for internal learning. This new briefing describes how Star data can be used to improve service delivery.
Learning from Star data at all levels of the organisation
Over three-quarters of Outcomes Star users in our client survey said Star data reports were ‘useful for learning how their service was ‘doing’ and ‘helpful in managing or developing the ‘service’. Indeed, Star data can provide meaningful management information at all levels, from a service manager reviewing a single ‘worker’s caseload to a senior management team reviewing data aggregated across services.
Alongside other data (e.g. satisfaction surveys, output and process data), Star data reports, such as those available from our upgraded Star Online System, allow organisations to ask increasingly focused questions about what is happening with the people they support.
Managers can gain essential insights by looking at differences in starting points and change across outcome areas, client groups, and service settings. Because these insights are likely to be greatest when compared against prior expectations, Triangle has produced resources to support ‘Star data ‘forecasting’.
Learning from Initial Star readings
The distribution of first Star readings provides a valuable overview of people’s needs coming into the service. Star readings can be compared against expectations to ensure that service users are entering the service appropriately and are offered suitable interventions.
An excellent example of the use of first readings is in Staffordshire County Council, where they look at start readings to see if the families are at the right level of service. In our interview with the Commissioning Manager at the time, she told us that “if we have families in our Family Intervention service that have readings of five, I look a bit deeper to see if we’re really using our resources correctly”.
Learning from change in Star readings
Movement in Star readings for each outcome area also provides an opportunity to learn where things are going well and when further exploration of service delivery may be warranted.
For example, if one service shows different outcomes to another service, this is a starting point for further investigation:
Is there other evidence that one service facilitates better outcomes than another?
Are there reasons why one service might be supporting people better than another?
Is the service user profile different in the different services?
Is practice significantly different in that service, and might there be lessons for other services?
A more in-depth analysis of the movement from each Journey of Change stage is also possible, offering more significant potential for learning than typical numerical outcome scales. Managers can explore which stage transitions are happening frequently and where there may be blockages to making other transitions. For example, a service may be very good at helping service users to begin accepting help but struggle more with moving them towards greater self-reliance, limiting the progress currently being made. Specific changes to service delivery might then need to be developed.
This is a new and improved edition, drawing on independent research and feedback from service users, keyworkers and organisations.
The new edition retains the person-centred, strengths based approach of previous editions but with even more accessible language, incorporating trauma-informed thinking and fuller acknowledgement of the impact of external factors.
There is fuller recognition of the necessity of on-going support for enduring and severe conditions. It is backed by a report on independent research into the psychometric properties and a review of literature supporting the Journey of Change and choice of outcome areas.
Both Stars were launched at the Govconnect Mental Health 2019 Conference at the Royal Society of Medicine on 26th September.
If you have any questions about our new Stars, any queries about transitioning between the Recovery Star 3rd Edition and the new Edition, or you would just like find out more about how the Stars can support your service users, keyworkers and organisation, please contact us on email@example.com or +44 (0) 207 272 8765.
The Outcomes Star has been tested psychometrically. A new set of psychometric factsheets demonstrate the validity of the Outcomes Star, and reveal how the Star can produce informative and valuable outcomes data for commissioners, funders and organisations.
Psychometric testing tells us how confident we can be in the data produced by a measurement tool including whether it measures what it claims to measure and produces consistent scores.
Triangle has published a set of factsheets to demonstrate the psychometric properties of every version of the Star. We are also in the process of having an article validating the Family Star Plus published in a peer reviewed journal. Dr Anna Good has produced a psychometric factsheet for each of the Outcomes Stars, providing the findings from a number of these tests. She explains a bit more about the process and importance of the ensuring the Stars are tested psychometrically.
“At its essence, validity means that the information yielded by a test is appropriate, meaningful, and useful for decision making” (Osterlind, 2010, p. 89).
Psychometric validation has been used in some form for over a hundred years. It involves tests of validity (usefulness and meaningfulness) and reliability (consistency), for example:
expert opinion about the content of the measure
clustering of ‘items’ or questions into underlying constructs
consistency across the readings produced by each item
consistency across ‘raters’ using a tool
sensitivity to detect change over time
correlation with, or predicts of, other relevant outcomes
Why is it important to test the Star psychometrically? What are the benefits of testing the Outcomes Star? What’s the background to the research? Triangle recognises the importance of having ‘evidence and theory support the interpretations of test scores’ (AERA, APA & NCME, 1999, p.9), both because we are committed to creating scientifically sound and useful tools and because policy advisors, commissioners and managers require validated outcomes measures and want assurance of a rigorous process of development and testing.
The validation process is an important part of the development of new versions of the Star – we need to know that the outcome areas hang together coherently, whether any outcome areas are unnecessary because of overlap with other areas or have readings that cluster at one end of the Journey of Change.
Once there is sufficient data, we also conduct more extensive psychometric testing using data routinely collected using the published version of the Star. This is beneficial for demonstrating that the Star is responsive to change and that Star readings relate to other outcome measures, which is important both within Triangle and for evidencing the value of our tools externally.
What was involved in producing the psychometric factsheets? The initial validation work for new Stars is conducted using data from collaborators working with Triangle during the Star development and piloting process. It involves collecting Star readings and asking service users and keyworkers to complete questionnaires about the acceptability and how well the Star captures services users’ situations and needs.
The further testing of the published version uses a larger sample size of routinely collected Star data and assesses the sensitivity of the Star in detecting change occurring during engagement with services. Whenever possible, we collaborate with organisations to assess the relationship between Star readings and validated measures or ‘hard outcome measures’ such as school attendance.
We have also been working to assess consistency in worker’s understanding of the scales using a case study method. This method is described fully in an article published in the Journal of Housing Care and Support (MacKeith, 2014), but essentially involves working with organisations using the Star to develop an anonymised case study or ‘service user profile’, and comparing the readings assigned by trained workers with those agreed by a panel of Star experts. The findings tell us how consistent and accurate workers in applying the Star scales when given the same information.
Conclusion: An evidence-based tool The Outcomes Star is an evidence-based tool. The development of new Stars follows a standardized and systematic process of evidence gathering through literature reviews, focus groups, refinement, initial psychometric analyses and full psychometric testing using routinely collected data.
Psychometric validation is useful in the development of new Stars and to provide evidence that the Outcome Star can produce data that meaningfully reflects the construct it is designed to measure.
Organisations can use Triangle’s psychometric factsheets alongside peer reviewed articles to demonstrate the validity of the Outcomes Star to funders and commissioners, and to have confidence that provided it is implemented well, the Star can produce informative and useful data.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing (4th ed.). Washington, DC: American Educational Research Association.
Mackeith, J. (2014). Assessing the reliability of the Outcomes Star in research and practice. Housing, Care and Support, 17(4), 188-197.
Osterlind, S. J. (2010). Modern measurement: Theory, principles, and applications of mental appraisal (2nd ed.). Boston, MA: Pearson Education.
Dr Anna Good: Dr Anna Good is a Research Analyst at Triangle: a large part of her role involves testing the psychometric properties of the Star, conducting research and supporting organisations to make the best use of Star data. After completing an MSc and a PhD in Psychology with specialisms in behaviour change interventions and psychological research methods, Anna spent a number of years as a post-doctoral researcher, including two years as principal investigator on a prestigious grant examining health behaviour change.