The Subjectivity of Focus Groups and Ink Blot Tests
Companies need to understand how people see their products and what they want, so asking customers about this in a focus group seems like a great idea. Focus groups can provide a valuable form of qualitative research, giving companies insight into consumers’ beliefs, desires, and attitudes surrounding a product. However, while focus groups can provide some insight, the history of projective tests (aka ink blot tests) in psychology offer a cautionary tale on solely relying on this kind of self-reported qualitative data.
Beginning in the early 20th century psychologists and psychiatrists developed projective tests to diagnose mental disorders and gain access to patients’ unconscious beliefs and desires. These tests, based on Freud’s theory of projection, were thought to allow unconscious beliefs and desires to surface through their open-ended structure, which was believed to be less threatening to people. In a projective test, someone is shown a set of ambiguous or abstract images that can be interpreted in many ways (the most famous example is the Rorschach ink blots, commonly portrayed in psychological examinations in movies), and they are asked to talk about what they see and what the images make them think of. It was believed that people will project their subconscious thoughts (desires, beliefs, etc.) onto the image, thereby revealing hidden parts of their personality that could then be analyzed and interpreted by the psychiatrist administering the test.
Projective tests were heralded as a breakthrough diagnostic tool, until they came under scrutiny in the 1980s and 90s, as psychoanalytic theory fell out of favor. The problem with projective tests is that they lack validity and reliability, the two critical aspects of any psychological assessment. Reliability refers to how consistent the results of a given test are: a test that is reliable will yield the same results time and time again. Validity refers to whether something actually measures what it claims to be measuring. Projective tests are unreliable for two reasons. First, it is unclear whether what people say actually reflects anything meaningful about their psyche. Sometimes it might, often it doesn’t, and it is often impossible to tell the difference. Second, even if someone’s responses do reflect something meaningful, the results are subject to the interpretation of the examiner. Research on projective tests has shown that interpretations of the exact same data vary widely, and thus these tests yield different results depending on who is interpreting them. A test cannot be valid if it is not reliable (reliability is a necessary, but not sufficient condition for validity), but validity is further compromised since it isn’t clear what subjects’ reports actually even reveal about their psyche. For these reasons, serious clinicians no longer use projective tests for diagnosis, but only as icebreakers to get a conversation started.
In many ways, focus groups resemble projective tests. People are asked to interact with something they’ve often never seen before and comment on it in some way (akin to the role of a patient in a projective test). Then one or more observers interpret their responses (akin to the role of the psychiatrist in a projective test). Focus groups suffer from the same issues with reliability and validity as projective tests. Because people are often unaware of critical aspects of their own minds, it is hard to tell whether their reports truly reflect meaningful insights into their decision-making. Furthermore, the interpretation of the results crucially depends on who’s doing the interpreting. As anyone who has worked with focus groups knows, the interpretation of the results often varies widely across observers, as observers’ own agendas and beliefs are inevitably incorporated into what they think people’s reports mean. Projection is a two way street, and often observers’ conclusions reveal more about their own biases than about what the respondents actually said.
This is not to say that focus groups provide no value, even such subjective qualitative data is better than no data, but it should be clear that information collected and analyzed from focus groups must be taken with a grain of salt. They can provide good icebreakers, and help inform how people might think about a product, but it is hard to know whether their results are valid, or their conclusions are reliable. In our opinion any such results must be confirmed by further research using objective, quantitative methods, with known validity and reliability.
(Co-authored by Kyle Thomas.)
