Shedding light on how health policies are implemented
Project title: Policy and program implementation and the role of context in explaining prevention effectiveness
Start date: January 2016
Expected end date: April 2020
What is the issue?
Policies and programs that can help prevent chronic disease sometimes fall short of success because they are not fully implemented or don’t reach the right people. Electronic methods for implementation monitoring and performance assessment are becoming more widely used. But do they tell the whole story? Can they explain why it is harder to achieve optimal implementation in some places but not in others?
This project explored some state-of-the-art electronic methods for collecting data about health prevention policy and program distribution; and to examine and quantify the extent to which resources and organisational capacity contribute to implementing policies and achieving targets. We aim to collate practitioner insights for ongoing practice and e-monitoring improvement.
How did the project address the issue?
The project team established a collaborative partnership with NSW Health to identify examples of monitoring systems that can help answer the following questions:
- How are policies and programs reflected in the e-monitoring of health promotion?
- What are the current best practices?
- What variation in policy and program implementation is captured across different geographic areas and contexts?
- How is policy and program “dose” or intensity defined and measured?
- What parts of the implementation and health promotion practice story are we possibly missing in current electronic monitoring systems? How can these stories be uncovered and brought to light?
The team used qualitative methods (ethnography and interviews) and quantitative methods (social network analysis) to identify and quantify factors across different regional contexts that explain experience and variation in the implementation of programs of policies.
Our case study centred around the Population Health Information and Management System (PHIMS). This IT system enables NSW Health to set up, manage and report against the aims of the Healthy Children Initiative, which targets early childcare and primary schools in NSW to promote healthy eating and physical activity.
In addition to working with the health sector, the team wanted to identify examples from other sectors – such as education, justice and community services – that could help inform best practices for health policy and practice monitoring and implementation.
Relevance for practice
The project provided insights to enhance the scope and sensitivity of methods used to track policy and program distribution, thereby making the tracking systems (for accountability) more accurate and useful. It will increase the likelihood that prevention will be more effective and sustained by making practice more visible and supported.
What were the outcomes?
The data gathered during this project are informing the development of electronic program monitoring systems in NSW and are expected to lead to improvements to the system.
For more information on this project, please visit our external website.
Professor Penny Hawe, Menzies Centre for Health Policy, The University of Sydney
- Dr Dan Chamberlain, La Trobe University
- Dr Katie Conte, Menzies Centre for Health Policy, The University of Sydney
- Dr Sisse Groen, Menzies Centre for Health Policy, The University of Sydney
- Ms Christine Innes-Hughes, NSW Office of Preventive Health
- Dr Victoria Loblay, Menzies Centre for Health Policy, The University of Sydney
- Ms Amanda Lockeridge, NSW Office of Preventive Health
- Leah Marks, University of Sydney
- Dr Andrew Milat, Centre for Epidemiology and Evidence, NSW Ministry of Health
- Dr Jo Mitchell, Centre for Population Health, NSW Ministry of Health
- Ms Lina Persson, Centre for Epidemiology and Evidence, NSW Ministry of Health
- Dr Samantha Rowbotham, Menzies Centre for Health Policy, The University of Sydney
- Abeera Shahid, McMaster University, Canada (internship)
- Associate Professor Sarah Thackway, Centre for Epidemiology and Evidence, NSW Ministry of Health
- Ms Mandy Williams, Health Promotion Service, South Western Sydney Local Health District
This project was funded by the NHMRC, Australian Government Department of Health, NSW Ministry of Health, ACT Health and the HCF Research Foundation.
- We have developed a website, including newsletters and webinars
- A number of papers are under review or being prepared
- The project findings will be disseminated in 2019
- Conte K, Marks L, Loblay V, Grøn S, Green A, et al. Can an electronic monitoring system capture implementation of health promotion programs? A focussed ethnographic exploration of the story behind program monitoring data. BMC Public Health. 2020 Jun 12;20(1):917. doi: 10.1186/s12889-020-08644-2.
- Conte K P, Davidson S. Using a ‘rich picture’ to facilitate systems thinking in research coproduction. Health Research Policy and Systems 2020:18(14).
- Loblay V, Conte KP, Grøn S, Green A, Innes-Hughes C, Milat A, Persson L, Williams M, Mitchell J, Hawe P. The weight of words: Co-analysis of thick ethnographic description and “friction” as methodological strategies in a health policy research partnership. Qualitative Health Research. 2020:1049732320962438
- Rowbotham S, Conte K, Hawe P. Variation in the operationalisation of dose in implementation of health promotion interventions: insights and recommendations from a scoping review. Implement Sci. 2019;14:56. doi: 10.1186/s13012-019-0899-x
- Conte KP, Shahid A, Grøn S, Loblay V, Green A, Innes-Hughes C, Milat A, Persson L, Williams M, Thackway S, Mitchell J, Hawe P. Capturing implementation knowledge: applying focused ethnography to study how implementers generate and manage knowledge in the scale-up of obesity prevention programs. Implement Sci. 2019;14:91.
- Hawe P, Conte K.P, Groen S, Loblay V, Green A, Innes-Hughes C, Milat A, Persson L, Mitchell J, Thackway S, Williams M. Mock abstracts with mock findings: a device to catalyse production, interpretation and use of knowledge outputs in a university policy-practice-research partnership. Evid Policy. 2019 Oct 09. Online. Available from: https://doi.org/10.1332/174426419X15679623018185
- Grøn S, Loblay V, Conte KP, Green A, Innes-Hughes C, Milat A, Mitchell J, Persson L, Thackway S, Williams M, Hawe P. Key performance indicators for program scale-up and divergent practice styles: A study from NSW, Australia. Health Promot Int. 2020:daaa001. doi: 10.1093/heapro/daaa001. [Epub ahead of print.]
Conte K, Hawe P. Will e-monitoring of policy and program implementation stifle or enhance practice? How would we Know? Frontiers of Public Health. doi:10.3389/fpubh.2018.00243
- Green AM, Innes-Hughes C, Rissel C, Mitchell J, Milat A, Williams M, Persson L, Thackway S, Lewis N, Wiggers J. Co-design of the Population Health Information Management System to measure reach and practice change of childhood obesity programs. Public Health Research and Practice (in press)
Conte KP, Groen S, Loblay V, Green A, Milat A, Persson L, Innes-Hughes C, Mitchell J, Thackway S, Williams M, Hawe P. Dynamics behind the scale up of evidence-based obesity prevention: protocol for a multi-site case study of an electronic implementation monitoring system in health promotion practice. Implement Sci. 2017; 12(146). doi: 10.1186/s13012-017-0686-5
- Loblay, V, Conte, KP, Groen, S, Green, A, Milat, A, Persson, L, Innes-Hughes, C, Mitchell, J, Thackway, S, Williams, M, Hawe, P. Data trails and documentation in Health Promotion services: sharing and storing practices in a professional context. Public Health Prevention Conference, Sydney, Australia, 2018.
- Conte, KP, Groen, S, Loblay, V, Green, A, Milat, A, Persson, L, Innes-Hughes, C, Mitchell, J, Thackway, S, Williams, M, Hawe, P. Challenging ‘pipeline’ translational models: quilting evidence in the largest-ever Australian childhood-obesity-prevention program. Public Health Prevention Conference, Sydney, Australia, 2018.
- Shahid, A, Conte, KP, Groen, S, Loblay, V, Green, A, Milat, A, Persson, L, Innes-Hughes, C, Mitchell, J, Thackway, S, Williams, M, Hawe, P. Informal knowledge management systems: a unique ethnography to help policy makers ‘listen close’ to public health practitioners monitoring their practice” Fuse International Conference on Knowledge Exchange in Public Health, Vancouver, Canada, 2018.
- Goldberg, E, Conte, KP, Groen, S, Loblay, V, Green, A, Milat, A, Persson, L, Innes-Hughes, C, Mitchell, J, Thackway, S, Williams, M, Hawe, P. How electronic monitoring of program implementation becomes embedded in health promotion practice. Public Health Prevention Conference, Sydney, Australia, 2018.
- Loblay V. Co-production and consensus: doing ethnography in a collaborative research partnership. Society for the Social Studies of Science. Sydney 29 August-1 September 2018
- Hawe P. What types of evidence are useful to understand interventions in complex systems. Global Evidence and Implementation Summit. Melbourne 22-24 October 2018
- Lobley, V. Resources and relationships: The complex role of resources in the implementation of childhood obesity interventions. Emerging Health Policy Research Conference; 27 July 2017, Sydney.
- Groen, S. When can we tick the box? The story behind achieving the hardest KPI in a health promotion program. Emerging Health Policy Research Conference; 27 July 2017, Sydney.
- Conte, KP, Gwynn, J, Turner, N, Koller, C, Gilham, K. Engaging local Aboriginal lay leaders to deliver health promotion programs to remote and rural communities. Poche Centre for Indigenous Health Research Showcase; 28 July 2016, Sydney.
- Conte, KP, Hawe, P. What’s measured and what’s not: dynamics of program implementation monitoring. Emerging Health Policy Research Conference; 13 July 2016, Sydney.
- Rowbotham, S. Conte, KP, Hawe, P. Operationalising ‘dose’ in public health interventions: A scoping review. Emerging Health Policy Research Conference; 13 July 2016, Sydney.
- Prevention Centre news, July 2016: Prevention Centre research shines at Menzies conference
- Prevention Centre blog, August 2017: Studying how IT monitoring affects prevention delivery in NSW
Scaling-up complex interventions: Adaptation is not a threat to fidelity (and maybe never was) Professor Penny Hawe, 14 August 2020, University of Sydney
- Panel discussion on researchers and decision-makers partnering to prevent chronic disease (Dr Jo Mitchell talks about the project around 48:00 mark)