How Representative are Volunteers?

As if by magic, another item at the BPS Research Digest which is also relevant to my recent forays discusses the question of whether participants in psychology studies are "representative" of the total sample under review. It seems like the majority of those who take part in psychology studies are generally more "stable and outgoing", which begs questions about whether said studies are reliable in their testing of depression measures, for example.

To give some background, the popular five-factor model measures personalities in terms of five separate factors (known as the "Big 5" model): Neuroticism, Conscientiousness, Agreeableness, Extraversion and Psychoticism. A depressed person, for instance, might have a higher Neuroticism (or even Psychoticism!) measurement than most people and a lower Extraversion rating. On the opposite end, an outgoing and popular person is likely to have high ratings for Extraversion, Agreeableness, perhaps even Conscientiousness. It is easy to see how personalities can be generally rated and measured according to these five factors.

[And for those who think there is much more to personality than these five categories, yes, you would be right, but statistical factor analysis has shown that most personality quirks come under one of the five umbrella factors of the Five-Factor model. Other personality models exist, of course, such as the 16PF Model.]

A study by Lonnqvist et al. (2007) mailed 61 military officers a survey on values (out of the 158 who were originally approached) who had completed personality assessments three years previously as per army recruitment procedures. The results showed that the respondents evidenced lower measures of Neuroticism and higher measures of Conscientiousness, Agreeableness and Extraversion. That's just about what you might expect from a military officer, don't you think? Although I'm only going by the BPS Review, it is reported that measures were established as opposed to those who didn't return the survey. How do they know? Just because 97 officers didn't bother to return the survey doesn't mean that they evidence higher measures of Neuroticism, or even that they have lower measures of Conscientiousness, Agreeableness and Extraversion!

And what about gender differences? All of these military officers were male, could there have been a difference if females were included in the study? And above all, is it appropriate to make a massive generalisation about all or most psychology participants based on a sample of 61 military officers? Don't participants come from different strata of society and from all walks of life? I did a post-hoc power analysis to check the sample size and even though this study seems more than sufficiently powered (.99!), the way that simple criticisms like mine above don't seem to have been properly addressed doesn't exactly scream of reliability to me.

It would make sense, however, if this was some sort of pilot study and a larger and more inclusive study is in the works, but no mention of such has been made.

In any case, a second study (by the same researchers) also used a survey approach: siblings from 15 families assessed the personalities of their brothers or sisters and also asked to volunteer for further tests and interviews. The subsequent sibling ratings showed that 55 participants who volunteered for further testing scored higher for Conscientiousness, Extraversion and Agreeableness than the 29 who declined the option of further testing. This is consistent with the study of military officers, and leads to the general finding by Lonnqvist et al. that volunteers for pyschological studies "are ... better adjusted than nonvolunteers. More specifically, those who are willing to volunteer as research participants tend to be lower in Neuroticism and higher in Conscientiousness than are those who are reluctant to volunteer."

Despite the simple criticisms, the study is reliable enough to be considered carefully since it has implications for psychological research. It means that there is a reasonable-to-good chance that participants who choose to volunteer for studies, who have low Neuroticism levels, are more likely to exhibit positive responses to drug treatments for depression or other things like panic disorders. But more importantly, the results of personality questionnaires are likely to be skewed because of the likely preponderance of volunteers who are not really representative of the full range of Five Factors. This is certainly an issue to take into consideration.

So how to go about trying to find a representative sample? Lonnqvist et al. suggest "acquiring samples comprised of people who are required to serve as experimental participants (e.g. as part of their jobs or academic programs)." This is fair enough; in many universities it is a requirement for 1st-year undergrads to take part in studies of 3rd-year students to acquire course credits. This can be a good thing, but my experience is that such students usually do these things in a rush and are focused more on obtaining the course credits than conscientiously (pun intended) participating in the study. Also, undergraduate students aren't always the sample of choice for most studies.

Another recommendation is that the research should be presented in an "attractive" way in order to attract a wide range of people to take part. This would be a reasonable proposal since the mere increase in sample size may simply mean an increase in volunteers, yet efforts to test a wider range of people would be more likely to yield representative results. This isn't a very solid proposal though, in my opinion. A final recommendation was to "attempt to evaluate the representativeness of the volunteer sample against the relevant population on the variables of interest." Huh? Comparing the sample to the population, basically? Isn't the point of all psychological studies to extrapolate the findings to the general population? So in other words, there really isn't anything (much) you can do to make your sample more representative. Just try and present your study in a more "attractive" way, whatever that means.

I have a tip: Just be nice to people. Talk to them and inform them that you're doing a psychology study, and ask them politely if they'd like to help you by taking part. If you've caught them at a good time, chances are they'll agree to participate. And this isn't necessarily an indication of volunteering. Some participants may find it something of a cathartic experience to complete the BDI-II questionnaire, for example. At least, those I've tested anyway.

________________________

Lönnqvist, J., Paunonen, S., Verkasalo, M., Leikas, S., Tuulio-Henriksson, A., Lönnqvist, J. (2007). Personality characteristics of research volunteers. European Journal of Personality, 21(8), 1017-1030. DOI: 10.1002/per.655