We have started to share outcomes data with commissioners, including Supporting People, Social Services and the Primary Care Trust, and have had very positive responses. One said they wished they had this kind of information for other services they commission.

Croftlands Trust

Validity of the data

The development of the Outcomes Star™ has been a bottom-up process in which the developers have collaborated with workers, managers, service users and sector bodies to test, improve and re-test the tool. More information on the development and testing of the tool.

The result is a tool with high levels of face validity and practical validity including:

Meaningful information: the tool demonstrates change in a meaningful way – i.e. some clients show positive change and some negative, some clients show large amounts of change and some small amounts. There is a variation in starting points and finishing points across clients and there are recognisable patterns of difference across different client groups (e.g. women compared with men). Download St Mungo's Outcomes Star™ Briefing for an example of the way one homelessness organisation has used its Outcomes Star™ data for learning.

Usability for staff: the tool makes sense to staff, they understand it and it fits well with and often improves established ways of working. For example in a study of 25 organisations using the homelessness version of the Star all 25 reported that it had improved keywork.

Usability for service users: clients are prepared to use it, they understand it and find that it helps them to recognise the positive progress they have made and plan next steps. For example, research among mental health services users piloting the Recovery Star found that two-thirds considered the tool to be helpful and to be a useful reflection of where they were and the progress made. The other third said it was no worse than other forms of assessment.

Usability for managers: the tool produces information which is useful for managers in evaluating the service, identifying strengths and areas needing attention. In addition managers have reported that the Star can provide a helpful basis for reviewing progress and next steps in team-meetings and supervision.

More information on the usability for managers, staff and service users.

We are currently exploring the scientific validation of the homelessness and mental health versions of the Outcomes Star™. This is likely to involve looking at test-re-test consistency of scoring and inter-rater reliability. We are also interested in scientific validation of the predictive validity of the scores e.g. the extent to which progress on the Star and reaching certain scores on the Star correlates with hard outcomes achieved.

Those wanting academic references may be interested to know that the model which underpins the homelessness and mental health versions maps closely on to the Cycle of Change which is widely used in the addiction field (Prochaska and DiClemente, C. C. (1982) Transtheoretical Therapy: Towards a more integrative model of change” Psychotherapy: Therapy, Research and Practice, Vol. 19 pp 276-88). The use of the model and the outcome areas used in the mental health version of the Star was informed by a review of user accounts of recovery (Anderson, R. Oades, L. And Caputi, P. (2003) Australian and New Zealand Journal of Psychiatry, volume 37, number 5, pp 586-594 “The experience of recovery from schizophrenia: towards an empirically validated stage model”.