The England Innovations Programme (EIP) has published the evaluation reports on the MTM Signs of Safety project with 10 local authorities (Baginsky et.al 2020) and the summary evaluation report on all EIP projects (FitzSimons and McCracken 2020).
Evaluation of MTM’s Signs of Safety Pilots – Evaluation report
Evaluation of MTM’s Signs of Safety Pilots – Evaluation report appendices
The reports conclude that the MTM project had no impact. MTM accepts that the outcomes we set ourselves — transformation of the organisations and their child protection case results — were not matched by our interventions.
A more nuanced picture of the MTM EIP impacts is reflected through an analysis of Ofsted reports and the surveys of staff and families in the local authorities (Munro and Turnell 2020). Some did very well and others made no progress or in fact went backwards. Whereas these mixed results wash out to show no impact for the whole set of local authorities, Munro and Turnell demonstrate that the results depended on what the local authorities actually did, particularly in their leadership and the extent to which they aligned organisational systems with the practice.
MTM takes this as a sobering reminder that a poor implementation of Signs of Safety could make things worse. This should be no surprise as introducing another set of demands into a busy and complex environment without clear commitment and making it fit and thus allowing space for the work, will produce as much contention as traction.
MTM also notes that the Partner in Practice Lincolnshire project that included embedding Signs of Safety and the Partner in Practice Richmond and Kingston Achieving for Children project that included implementing Signs of Safety both showed “some positive impact”. Additionally, the key messages from the summary evaluation report on all EIP projects (FitzSimons and McCracken 2020) describing what is effective in practice accord with the key features of the Signs of Safety model.
The evidence from both the independent study and MTM’s own (Baginsky et al 2020 and Munro and Turnell 2020) shows that Signs of Safety was only partially and to varying degrees implemented in the ten local authorities and so the studies tell us about the challenges of implementation not the impact on families of receiving a Signs of Safety service.
“While MTM is confident that Signs of Safety can make powerful differences for particular cases, particular practitioners and teams, we take lessons from the research as to the multiple factors that affect adoption of Signs of Safety and the difficulties of achieving the desired depth and scale of practice in a sustainable way.”
The research is testament to the assertion by Pawson (2006) that “social interventions (such a practice framework) are complex systems thrust into complex systems”. While MTM is confident that Signs of Safety can make powerful differences for particular cases, particular practitioners and teams, we take lessons from the research as to the multiple factors that affect adoption of Signs of Safety and the difficulties of achieving the desired depth and scale of practice in a sustainable way.
MTM takes strong note of reported confusion among some staff about what the Signs of Safety actually is. It accepts that as Signs of Safety is a comprehensive framework and methodology for child protection practice comprising strong principles, practice methods and disciplines for applying those methods, setting a process for the practice rather than prescribing the content, there is a lot to grasp. Moreover, MTM recognises the challenge, as Baginsky et al (2020) reported: “Supporting families to take responsibility, and to work in partnership with them to do so, is at the heart of Signs of Safety. While it was viewed as a strength of the model it was not always seen to be compatible with statutory social work in England or with the high level of risk involved in many child protections cases. It was agreed that MTM could have given more guidance on how to achieve this using Signs of Safety. The concern was that Signs of Safety in the hands of an experienced child protection social worker was very different from that used by an inexperienced worker, where it could become superficial and the line between the model and statutory responsibilities become blurred.”
Addressing both the clarity of the practice and implementation issues is imperative as Baginsky et al (2020) have reported that around a third of local authorities in England use Signs of Safety as their model of practice and another third also do so in combination with other approaches.
In offering the Signs of Safety, MTM considers that we come to the complex systems of child protection agencies with values, aspirations and a vision of collaborative practice and partnership with families that the vast majority of child protection professionals want to embrace. We offer methods and tools within a framework for this practice. Additionally we look to get alongside agencies and their leadership to work on whole system change that involves detailed attention and slow thinking and discerning judgement. While MTM has a well-developed implementation framework with good methods, and some local authorities did well in the EIP, it unreservedly accepts that it has not sorted out all the implementation issues and is on a journey with child protection agencies in doing so. MTM’s strongest challenges lie in building a shared understanding and working in partnership with agencies and researchers of how significant the paradigm shift is, how complex the systems are that we are trying to change, how hard it is to do so, and how likely failure always is.
MTM and the the Signs of Safety community has learned an enormous amount from the England Innovation Project. The Signs of Safety practice has evolved in the key areas of analysis and safety planning, the implementation framework has been refined and further developed to be more focussed and include practical methods that support the agency to own the approach, measures for practice breadth, depth and impact have been developed, and the Signs of Safety IT Solution that fit with agencies’ existing systems to deliver essential alignment and substantial efficiencies is available.
Profound change can be achieved. Some EIP local authorities did so. And another recent Ofsted assessment of “outstanding” for North Tyneside, having worked on whole system implementation over some years including a “learning lab” for the Signs of Safety IT Solution, attests:
The way in which the local authority’s preferred method of social work has been rolled out across early help and children’s social care, and embraced by partners, has had a transformational impact. It provides a common language with which to talk about and explore issues and concerns, needs and risks, dangers and protective factors in a way that is easy to understand for parents, professionals and partners. Particularly impressive is the way in which the local authority’s electronic case recording system has been adapted to ensure that it helps rather than hinders this approach. Equally impressive is the way in which senior leaders are leading by example, using the same simple methodology in reports and policy documents.
A comprehensive summary of Signs of Safety research on family outcomes and parent perceptions (Pecora 2020) is available on the Signs of Safety Knowledge Bank.
Professor Eileen Munro and Professor Andrew Turnell with Marie Devine and Dr Jack Cunliffe, You Can’t Grow Roses In Concrete Part 2 Action Research Final Report, Signs of Safety England Innovation Programme, Elia International Ltd, 2020
Mary Baginsky, Ben Hickman, Jess Harris, Jill Manthorpe, Michael Sanders, Aoife O’Higgins, Eva Schoenwald and Vicky Clayton, Evaluation of MTM’s Signs of Safety Pilots Evaluation report, October 2020. UK Department for Education
Dr Ana FitzSimons and Katie McCracken, Opcit Research Children’s Social Care Innovation Programme, Round 2 Final Report, September 2020, UK Department for Education
Professor Peter Pecora, Signs of Safety Research Summary of Family Outcomes and Parent Perceptions, 2020, Casey Family Progams