Search eceee proceedings

Reflections on representativeness of voluntarily recruited intervention participants compared to a unique nation-wide data set

Panel: 4. Monitoring and evaluation for greater impact

Authors:
Devon Wemyss, ZHAW Zurich University of Applied Sciences, Switzerland
Yann Blumer, Zurich University of Applied Sciences

Abstract

Many energy-related intervention studies use voluntary recruitment of participants. Often findings from these studies are compared to a control group or the general population. However, typically only socio-demographic data is available for these comparisons, thus it is not possible to know whether the intervention participants are representative in terms of their living context, attitudes, or social influences. This raises questions related to the generalizability and scalability of the findings.

In 2016, as part of a Swiss energy research programme, a nation-wide panel was established to collect data on energy use behaviours in Swiss households. Importantly, contextual and psychological variables were collected, along with socio-demographic information. This data is compared to items measured in baseline studies of three energy behaviour studies in Switzerland (details in slides). The three Swiss studies aimed at: 1) household electricity savings through a neighbourhood challenge, 2) replacing the car with an e-bike, and 3) using alternate modes of transport for sports teams. The results of the studies suggested that all three interventions were effective in reducing energy consumption.

As it was not possible to compare all four data sets across items beyond socio-demographic variables only individual studies were compared to the national dataset where possible. There were clear differences of socio-demographics between the interventions and the overall population, which can be explained by the study design (i.e. target group addressed). For example, in study two, which was aimed at car owners, participants had significantly more cars than the Swiss average. Similarly, study three focused on recreational sports teams and this resulted in a lower average age than the population.

Less intuitively, differences in environmental attitudes and social norms are not always in line with what could be expected considering the study contexts. For example, in study one, a neighbourhood challenge, the participants indicated that they feel less pressure from their family and friends to save energy (injunctive norm) as compared to the general population. But they did think that more people around them actively save energy (descriptive norm). Indeed, the study one participants felt significantly more obligated to save energy. The range of responses show that while the participants may be more environmentally conscious of their consumption in some areas, they nevertheless also exhibit typical ‘unsustainable’ behavioural patterns, similar to the general population.

This exploratory analysis highlights the challenges in comparing study impacts, and questions assumptions around the expected representativeness of voluntarily recruited participants. Comparing results is challenging and thus efforts to implement standardized questions, such as suggested by IEA-DSM Task 24, can make this exercise easier and help identify (more) effective schemes. However, specific behaviour is strongly embedded in the individual’s context, and thus reported behaviour may be better used as a proxy for awareness.

For interventions that target a very specific behaviour, choosing specific channels and programme framing can elicit a self-selection bias which can be beneficial for the effectiveness of the study (i.e. a mobility intervention that addresses heavy car use), albeit more challenging for researchers to compare results. On a more general level, this analysis highlights the need in energy social science research to reflect very carefully whether transferability or effectiveness is more important for scalability of the approach and generalizability of the results.

Downloads

Download this presentation as pdf: 4-039-19_Wemyss_Presentation.pdf