Autism researchers are control freaks. A large part of what we do is concerned with ruling out alternative explanations for our results, which involves designing carefully controlled experiments. Autism is a pervasive developmental disorder, meaning that it affects just about every aspect of cognition (and apparently most parts of the brain). So there are always plenty of alternative explanations to worry about. And attempting to publish your research involves running the gauntlet of reviewers, each primed with their own alternative explanations, which you hadn’t even considered let alone controlled for. Some of these may even be plausible.
In a new paper in Journal of Autism and Developmental Disorders, Sophie Lind and Dermot Bowler report a study looking at autistic children’s understanding of the concept that seeing something leads to knowing about it. This is an interesting topic because understanding the link between a person’s perception of the world and their consequent mental states is an important milestone in the development of theory of mind. And, of course, impaired theory of mind is widely argued to be the root cause of the social difficulties in autism. However, the study also highlights some important methodological issues relating to experimental control in autism research.
As Lind and Bowler point out, previous studies looking at the performance of autistic kids on "see-know" tests have had one major methodological problem or another. Either there have been floor effects (even the control group struggled on the task), or group differences have been confounded with differences in language ability. In one case, the statistical analyses and conclusions didn’t actually match up with the data reported. On top of this, it isn't really clear that failure on "see-know" tests is indicative of specific problems with comprehension of seeing and knowing. It could just be that kids with autism find these kinds of questions difficult in general.
To address these issues, the authors took a number of steps. They ensured that the tasks were age-appropriate to avoid floor effects and they compared the performance of children with autism to a control group matched on age and verbal ability, allowing them to rule out language difficulties as a potential explanation. Most importantly, they added control questions to ensure that the kids could actually do the task. These questions were similar in structure and format to the "see-know" questions. The idea being that, if kids with autism can pass the control questions but still fail the "see-know" questions, then we can rule out any alternative explanation that predicts that they should also fail the control questions.
Lind and Bowler’s "see-know" test was similar to those used in previous autism studies. It involved two Playmobil characters, John and Fiona, and a load of boxes. On each trial, one of the characters opened a box, the other looked inside, and the child was asked “Who knows what is in the box?” The control task involved scenarios that didn’t involve seeing and knowing but were similar in their structure and memory demands. For example, the child was told that Fiona cut her knee while John got muddy knees; they were then asked who had sore knees.
 |
| Are you thinking what I'm thinking? |
90% of the control children passed the see-know test, getting it right on at least 4 of the 5 trials. In comparison, only 60% of autistic children passed. However, children with autism were also much more likely to fail the control task.
The authors therefore excluded all the children who failed the control task, reasoning that their problems on the see-know task could be explained away in terms of “extraneous” task demands such as recalling the action sequences or making inferences in general. They then reanalysed the data, looking only at the children who passed the control task. There was still a statistically significant difference between the pass rates of the two groups, and so Lind and Bowler concluded that “children with ASD have a specific deficit in understanding that seeing leads to knowing”.
In many ways, this is a really neat study. Methodologically, it is certainly a cut above previous work in this area, for all the reasons mentioned earlier. The results seem pretty compelling - even controlling for all the extraneous task demands, there are still significant group differences in the pass rates. Case closed one might think. But, there is just one more thing…
Typically, researchers just exclude the participants who fail the control task and we hear no more about them. We’re left to assume that they would have failed the theory of mind task too. Helpfully, Lind and Bowler include the full cross-tabulated data showing the number of children in each group who passed and failed each task.
 |
| Table 2 from Lind & Bowler (2010) |
The kids who either pass both questions or fail both questions don't really tell us much. What we’re really interested in is the performance of kids in the off-diagonal cells of the tables – the ones who pass one task but fail the other. If autism is associated with specific difficulties understanding seeing and knowing, then we would expect to see lots of kids failing the see-know task despite passing the control task. In fact, only 8 out of the 40 kids with autism show this pattern (highlighted in the table). Three kids with autism actually show the opposite pattern. However, these kids were excluded in the final analysis. In the control group, the numbers are smaller but the ratio (2:1) is similar.
So, at the risk of sounding like one of those annoying reviewers I'm not entirely convinced that children with autism really do have a specific deficit in understanding that seeing leads to knowing. If anything, this reaffirms the importance of Lind and Bowler's novel control tasks. Further proof that, in autism research, control is everything.
Reference:
Lind, S., & Bowler, D. (2010). Impaired Performance on See-Know Tasks Amongst Children with Autism: Evidence of Specific Difficulties with Theory of Mind or Domain-General Task Factors? Journal of Autism and Developmental Disorders, 40 (4), 479-484 DOI: 10.1007/s10803-009-0889-y