Our September newsletter included updates on the Star Online system. We also introduced two new Outcomes Stars for mental health, the My Mind Star and the new and improved edition of the Recovery Star as well as updated research for the new edition of the Recovery Star™
Find out more
My Mind Star™ is an Outcomes Star for building and tracking well-being and resilience in young people and is for young people’s mental health and well-beining
The new and improved edition of the Recovery Star™: the Outcomes Star for mental health and well-being. This Star has been designed to support and measure progress towards recovery for adults experiencing mental health issues and contains changes to make the Star more appropriate, accessible and effective.
Joy MacKeith, Co-Founder and Director of Triangle, and one of the authors of the Outcomes Stars reflects on the Mental Health 2019 conference.
By Joy MacKeith, Director and Co-Founder of Triangle.
This was my second time at this conference bringing together senior NHS policy makers and service providers to chew over the state of the nation’s mental health. There certainly was a lot to think about….
Integration, integration, integration
The strongest message throughout the day
was to think holistically about mental health. Duncan Selbie, CEO of Public
Health England kicked things off by reminding us that the biggest determinants
of mental health are income, housing, work and social connection. This was then
driven home by Andrew Herd from the Department of Health and Social Care who is
engaged in the mammoth task of rallying all government departments around the
cause of mental health and inspiring, cajoling and sometimes requiring them to
factor mental health into their policy-making. It was refreshing to hear this
holistic drum being banged, music to my ears as one engaged in creating tools
which attempt to provide a map for the person’s whole life, not just the
problem that brought them through the door.
Young people on the top of the agenda
Most of the conversations that we had on our Outcomes Star stand were about My Mind Star – our newly published tool for early intervention with young people with mental health issues. There was real excitement about the strengths-based focus and intuitive visuals. The fact that it also provides service-wide distance travelled information was usually not the focus of the conversation, but nevertheless seen as an important plus. There is a big scale of up services for this group, Phoebe Robinson, Health of Mental Health for NHS England told us. That isn’t surprising given that a shocking one in three children and young people now have a diagnosable mental health issue.
Get ready for growth in social prescribing
GP, David Smart shared his experience of creating an integrated depression pathway in Northampton. What inspired me about this case study was seeing just how much difference can be made when people step out of their normal ways of doing things and look at a problem freshly. Social prescribing plays an important part in their approach and is an example of integration in action. It was mentioned often through the day as an approach that is on the rise. I was extra pleased when I heard that they are exploring the use of the Well-being Star to collaboratively assess needs and the measure outcomes of their social prescribing work.
Interested in finding out more about the Outcomes Stars for mental health and well-being?
The Recovery Star Fourth Edition is a new edition of the Outcomes Star for working with mental health and well-being. My Mind Star is the new Outcomes Star for young people young people’s mental health and well-being. Find out more here. To find out more about which which other Outcomes Stars are suitable for organisations working with mental health and well-being, please take a look at our sector page. For more information on the Outcomes Stars and our licensing and training options please contact us: email Triangle at email@example.com, or call on +44 (0) 202 7272 8765.
Joy MacKeith: Joy is a co-creator of the Outcomes Stars. She leads on innovation, data, research and the theory behind the Star. For more information on the evidence and research that underpins the Outcomes Stars visit our evidence and research library. Please contact us for any further information: email Triangle at firstname.lastname@example.org.
Our June newsletter included news on our upcoming new Star Online system, news on our new Outcomes Star: the New Mum’s Star, as well as updates on our new resources, reports and a short blog on our social enterprise model.
New resources and reports
Some of the new resources which were highlighted in the newsletter included:
Welsh translations of our Family Stars (available for licenced Star users on request)
A new evaluation report created with Chiltern Music Therapy examining the benefits of the Music Therapy Star which specifically highlighted that 95% of people engaging in music therapy significantly made improvements in at least two outcome areas.
Our March newsletter round-up includes updates on our licensing options, an introduction to our new Managing Director, and a call for interest and views on a Star for Neurodevelopment as well as updates on our new resources and reports.
New resources and reports
Triangle continues to publish new resources to support organisations in creating positive change by promoting an effective outcomes approach and providing sector-wide solutions:
This month’s newsletter round-up includes news on new reports; two new Outcomes Star case studies; new Star research and pyschometric findings; links to Family Nurse Partnership’s newly published blog post on the development of our upcoming New Mum Star and an interview with Clare Burrell, Head of Strategic Commissioning and Policy for Children and Families at Essex County Council on the Family Star (Relationships) the new Star for reducing inter-parental conflict.
New resources and reports
We have recently published a new set of psychometric factsheets as well as a short blog on the role of psychometric testing and why it matters. We also introduced new resources including:
The Family Star (Relationships) was one of two new Stars published by Triangle in September 2018. It is designed for services working with families where inter-parental conflict is a major focus, with the aim of protecting their children from the harmful effects and enabling them to thrive. We recently interviewed Clare Burrell, Head of Strategic Commissioning and Policy for Children and Families at Essex County Council, who was one of the key collaborators spearheading the development, to find out why they invested in the development of this new Outcomes Star and to hear her thoughts on Triangle’s collaborative process.
Clare, could you tell us a bit about your role and how you first came across the Outcomes Stars? I lead on Early Intervention where, like others, we take a holistic approach to our work. Five years ago, we set up our Family Innovation Fund which is delivered by voluntary and community providers across Essex. It is important for commissioners to understand and know that the right outcomes are being delivered and to evaluate the impact of the work. It was then we took the decision to ensure that all the providers of these services would use the same outcomes tools, so that we could measure apples with apples rather than apples with pears! The aim was that doing this would strengthen the case for what works and help us develop the economic case for early intervention. Our intendent evaluators, Traverse, reviewed the outcomes tools available and they recommended we adopt the Outcomes Stars. As a result we prescribed the Family Stars to our providers and over time, they have come to realise the potential and usefulness of them as tools.
How did your focus around inter-parental conflict develop? The primary outcome of our work was family stability and through our work with whole families, it became increasingly evident within the early intervention space that if we were going to improve outcomes for children, we needed to focus more sharply on parental relationships. At the same time, we became one of 11 Local Authorities working with the Department for Work & Pensions, to develop the evidence for what works to reduce inter-parental conflict. It was then that we partnered with colleagues at Hertfordshire County Council and with DWP, to commission an Outcomes Star that would support our work in this area. We needed something different to any of the Stars which already existed, to take in the specific intricacies of relationships.
Did you find anything challenging about the collaborative process to develop a Star? Inter-parental conflict is a relatively underdeveloped area of practice, so there was quite a bit of interesting discussion and some challenges for everyone to agree on the focus and content for the Star. A couple of providers decided not to continue with the process and pilot because the Star didn’t fit with their way of working with families – they didn’t have enough time with parents in conflict or were solely focused on getting an agreement for contact with children rather than working holistically, for example. It was helpful to have a provider with vast experience and knowledge of domestic abuse – they made an invaluable contribution to ensure that the Star didn’t inappropriately stray into that space and that it was clear where practitioners using the Star would draw the line.
What would your top tips be for anyone considering collaborating with Triangle to develop a Star? I would say there are two main areas. The first is getting the right people around the table. It can be hard at times, but it is important that the right voices and heads are in the room for the key Star development sessions. I focused on the ‘unusual suspects’ – people who came from all different perspectives but who would challenge and mix up our thinking. It was also important to get people who were committed to the whole of the process, as it takes a considerable amount of time – over a year in our case.
The second is to plan your pilot upfront and think about how you will get buy-in from practitioners. It’s more than just being in the sessions, it’s about committing to piloting the tool and feeding back throughout the process. We piloted with our commissioned providers, and some of our in-house services. This particular Star brought a challenge because it was used only as and when an inter-parental conflict focused case came up, so that limited the amount of data and feedback and made it more difficult to evaluate in a meaningful way, beyond case studies. That said, practitioners love the new Star and are using it in their work, including in unexpected circumstances – as an example one practitioner has recently used the Star to support a teenager and parent in conflict.
And lastly Clare, what are your hopes for the Family Star (Relationships)? My plan is that this Star is embedded in practice. Essex is a big place, but we are doing a lot to make sure that our practitioners know about this Star and that it is part of their toolkit to use with families and that they can draw on it when inter-parental conflict is the main issue for a family they support.
Many thanks to Clare for her time for this interview.
The Family Star (Relationships) is designed for practitioners working to identify and support the resolution of inter-parental conflict so that outcomes for children and young people are improved. It was developed by Triangle in 2017 – 2018 in collaboration with Essex County Council and Hertfordshire County Council. Please see the Family Stars page for more information about the new variant for supporting parents to manage conflict: the Family Star (Relationships).
94% of practitioners piloting the Star indicated they found the Family Star (Relationships) described the situation, strengths and needs of the service users they worked with either well (either fairly or very well). 80% of parents who completed feedback about the Family Star (Relationships) said it helped them understand what they needed in the way of support.¹ For more information about how Stars are developed, and Triangle’s collaborative process please see our Star Development page.
For an example of Star data in action, please see the recently published evaluation report from OPM Group for Essex County Council’s Early Help Programme, the Family Innovation Fund (FIF). FIF used the Family Star Plus and My Star as primary evaluation tools, with practitioners completing Stars in collaboration with service users, as an integrated part of the Early Help support being provided. Star data was used to demonstrate change, as part of an economic evaluation, and to make the case for early help provision.
The Outcomes Star has been tested psychometrically. A new set of psychometric factsheets demonstrate the validity of the Outcomes Star, and reveal how the Star can produce informative and valuable outcomes data for commissioners, funders and organisations.
Psychometric testing tells us how confident we can be in the data produced by a measurement tool including whether it measures what it claims to measure and produces consistent scores.
Triangle has published a set of factsheets to demonstrate the psychometric properties of every version of the Star. We are also in the process of having an article validating the Family Star Plus published in a peer reviewed journal. Dr Anna Good has produced a psychometric factsheet for each of the Outcomes Stars, providing the findings from a number of these tests. She explains a bit more about the process and importance of the ensuring the Stars are tested psychometrically.
“At its essence, validity means that the information yielded by a test is appropriate, meaningful, and useful for decision making” (Osterlind, 2010, p. 89).
Psychometric validation has been used in some form for over a hundred years. It involves tests of validity (usefulness and meaningfulness) and reliability (consistency), for example:
expert opinion about the content of the measure
clustering of ‘items’ or questions into underlying constructs
consistency across the readings produced by each item
consistency across ‘raters’ using a tool
sensitivity to detect change over time
correlation with, or predicts of, other relevant outcomes
Why is it important to test the Star psychometrically? What are the benefits of testing the Outcomes Star? What’s the background to the research? Triangle recognises the importance of having ‘evidence and theory support the interpretations of test scores’ (AERA, APA & NCME, 1999, p.9), both because we are committed to creating scientifically sound and useful tools and because policy advisors, commissioners and managers require validated outcomes measures and want assurance of a rigorous process of development and testing.
The validation process is an important part of the development of new versions of the Star – we need to know that the outcome areas hang together coherently, whether any outcome areas are unnecessary because of overlap with other areas or have readings that cluster at one end of the Journey of Change.
Once there is sufficient data, we also conduct more extensive psychometric testing using data routinely collected using the published version of the Star. This is beneficial for demonstrating that the Star is responsive to change and that Star readings relate to other outcome measures, which is important both within Triangle and for evidencing the value of our tools externally.
What was involved in producing the psychometric factsheets? The initial validation work for new Stars is conducted using data from collaborators working with Triangle during the Star development and piloting process. It involves collecting Star readings and asking service users and keyworkers to complete questionnaires about the acceptability and how well the Star captures services users’ situations and needs.
The further testing of the published version uses a larger sample size of routinely collected Star data and assesses the sensitivity of the Star in detecting change occurring during engagement with services. Whenever possible, we collaborate with organisations to assess the relationship between Star readings and validated measures or ‘hard outcome measures’ such as school attendance.
We have also been working to assess consistency in worker’s understanding of the scales using a case study method. This method is described fully in an article published in the Journal of Housing Care and Support (MacKeith, 2014), but essentially involves working with organisations using the Star to develop an anonymised case study or ‘service user profile’, and comparing the readings assigned by trained workers with those agreed by a panel of Star experts. The findings tell us how consistent and accurate workers in applying the Star scales when given the same information.
Conclusion: An evidence-based tool The Outcomes Star is an evidence-based tool. The development of new Stars follows a standardized and systematic process of evidence gathering through literature reviews, focus groups, refinement, initial psychometric analyses and full psychometric testing using routinely collected data.
Psychometric validation is useful in the development of new Stars and to provide evidence that the Outcome Star can produce data that meaningfully reflects the construct it is designed to measure.
Organisations can use Triangle’s psychometric factsheets alongside peer reviewed articles to demonstrate the validity of the Outcomes Star to funders and commissioners, and to have confidence that provided it is implemented well, the Star can produce informative and useful data.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing (4th ed.). Washington, DC: American Educational Research Association.
Mackeith, J. (2014). Assessing the reliability of the Outcomes Star in research and practice. Housing, Care and Support, 17(4), 188-197.
Osterlind, S. J. (2010). Modern measurement: Theory, principles, and applications of mental appraisal (2nd ed.). Boston, MA: Pearson Education.
Dr Anna Good: Dr Anna Good is a Research Analyst at Triangle: a large part of her role involves testing the psychometric properties of the Star, conducting research and supporting organisations to make the best use of Star data. After completing an MSc and a PhD in Psychology with specialisms in behaviour change interventions and psychological research methods, Anna spent a number of years as a post-doctoral researcher, including two years as principal investigator on a prestigious grant examining health behaviour change.
This month’s newsletter round-up includes news on two new Stars, newly published research on using Star data with research and evaluation in Early Help services, job vacancies, future Star development as well as upcoming events.
Outcomes Star events
Triangle are running free events across the country in Brighton, London and Glasgow for service managers and practitioners of frontline services. Morning sessions are for services who have heard about the Outcomes Star and would like to know more, afternoon sessions are for services who already use the and will focus on implementation and getting the most of out of the Star and data. Spaces are limited: find out more and book your place here.