The utility of natural experiments for improving health prevention evidence

I’m Dr Mel Crane from the University of Sydney and The Australian Prevention Partnership Centre to talk to you about natural experiments

Slide 1

A natural experiment is an experiment – that is, exposure and outcomes are analysed using methods that attempt to make causal inference HOWEVER in this situation the researcher has had no control over exposure to the event but uses the opportunity that is presented. So as an example The current COVID19 crisis has provided the opportunity for many researchers to undertake natural experiment studies to assess a variety of issues

Why are these studies necessary – because planned, controlled experiments like RCTs are rarely appropriate or possible in many public health evaluation situations.  Also, research evidence may be required quickly  – like a policy decision so we need research evidence that can accommodate these opportunistic research needs.  -a RCT study takes time to implement, and the opportunity may be over before the research study is set up.

Slide 2

SO the question is – how can natural experiments help to build evidence for obesity prevention ? Issues like obesity are complex, driven by multiple interrelated factors including environmental, social and cultural factors, not just individual-level behaviour. Prevention requires interventions that target multilevel systemic factors – such as taxation, restrictions on unhealthy food and healthy food availability as well as changes to the built environment, but evidence of the effectiveness of such interventions is small – limited by a real lack of evaluation evidence.

Slide 3

We conducted a research study to investigate the use of natural experiments for obesity prevention and to understand how they were being utilised. We searched peer-reviewed literature for studies that self-described as natural experiments and aimed at improving nutrition, physical activity or obesity specifically. I won’t go into the methods or the characteristics of interventions and evaluation methodologies. For more information, the paper has just been published or you can email me, melanie.crane@sydney.edu.au

Slide 4

The kind of information we were looking for to determine the utility of natural experiments was to determine the need for population health research evidence, as well as things like the ability for researcher manipulation of exposure conditions for research; ethical constraints and practical considerations that may prevent a planned researcher controlled study…

Slide 5

The purpose of the studies were largely to evaluate the effectiveness of an intervention or impact of a policy, with fewer studies aimed at exploring health inequalities or environmental/social determinants:

• A planned researcher driven research was not always feasible to answer the research question, however partial control over an intervention may be possible in some circumstances to collect baseline data but if a policy is enacted quickly then this would be restricted
• We note that studies aimed at assessing inequalities in the population would have been unethical as a planned experiment where there was risk of potential harm from intentionally restricting access to medical care, economic support, or randomising groups to receive social benefits or the freedom to migrate
• A number of practical considerations also necessitated the need for a natural experiment.

Slide 6

Natural experiments enable the evaluation of processes and outcomes of policies & interventions within the real-world complex social and political conditions they naturally operate. They offer opportunistic evidence where RCT methods may be impossible due to timing or exposure. They offer flexibility to use for a range of research questions, intervention settings and scale and evaluation methods – however, this increases the difficulty for making comparisons across studies.

They are often criticised for their inability to eliminate bias and bias due to confounding is a concern. We caution that deciding against natural experiments, as a result, ignores the contribution they can make to the overall evidence generation, particularly in regards to the complexity of real-world intervention evidence

But this can be reduced by ensuring good study design – such as having a control group comparator. We found less than one-third of the studies we assessed had a defined exposure and control group. Given these studies are often opportunistic, finding an unexposed group may be difficult. This is easier in environmental interventions where there is greater potential for researchers to be involved in establishing baseline data points. This is harder with a policy evaluation. Some studies have used modelling simulations to create a control group comparator, others employed step-wedge designs.

In response to the complexity of the obesity epidemic, obtaining policy and intervention impact evidence is critical and natural experiments may provide innovative translational research evidence on obesity policy interventions that have been missing.