Inferencing into the void: problems with implicit populations Comments on `Empirical software engineering experts on the use of students and professionals in experiments'

Abstract

I welcome the contribution from Falessi et al. [1] hereafter referred to as F++ , and the ensuing debate. Experimentation is an important tool within empirical software engineering, so how we select participants is clearly a relevant question. Moreover as F++ point out, the question is considerably more nuanced than the simple dichotomy it might appear to be at first sight. This commentary is structured as follows. In Section 2 I briefly summarise the arguments of F++ and comment on their approach. Next, in Section 3, I take a step back to consider the nature of representativeness in inferential arguments and the need for careful definition. Then I give three examples of using different types of participant to consider impact. I conclude by arguing, largely in agreement with F++, that the question of whether student participants are representative or not depends on the target population. However, we need to give careful consideration to defining that population and, in particular, not to overlook the representativeness of tasks and environment. This is facilitated by explicit description of the target populations.