The Adventures of DataThief

DataThief by Jed Pascoe: Reproduced with the artist's permission

Having spent much of the past week struggling to make sense of my data, it’s good to come home, pour a glass of wine, put on some Sharon Jones, and, er… play with somebody else’s data!

Recently, I’ve discovered DataThief - an application that allows you to scan in a graph from a paper and extract the data points. Sometimes, this provides insights that really aren’t obvious from the original paper.

The other week, for example, I came across an intriguing neuroimaging study reported on the SFARI website. In the study, Judith Verhoeven and colleagues used diffusion tensor MRI to examine the superior longitudinal fasciculus, a bundle of nerve fibres that is assumed (although see this paper) to connect two brain regions involved in language production and comprehension - Broca’s area (left front-ish) with Wernicke’s area (left and back a bit).

Verhoeven et al. reported that integrity of the superior longitudinal fasciculus was compromised in kids with specific language impairment (SLI) – that is, kids who have language difficulties for no obvious reason. However, the same was not true of kids with autism, even though they had poorer language skills than those with SLI.

Taken at face value, this is a pretty major blow to the idea that autism and SLI have anything more than a superficial resemblance [pdf].

DataThief, however, suggests otherwise.

The figure below is a scatterplot with each coloured shape representing a single child. On the x-axis is performance on a language test. On the y-axis is fractional anisotropy (FA) – the imaging measure used to assess the integrity of the left superior longitudinal fasciculus.

Figure 3a from Verhoeven et al 2011, showing integrity of the left superior longitudinal fasciculus plotted against the child's language scores (z-scores). Children with SLI in red, autistic kids are the blue squares. Control children are the green and blue circles. 

The purpose of the graph was to show the significant correlation between these two measures in the SLI group. But if we can read off the y-coordinates of each shape, we can show the distribution of fractional anisotropy scores for all three groups.

Cue DataThief.

It’s really just a case of clicking on three reference points for which you know the coordinates and then clicking on each of the data points in turn. Then you simply export the coordinates of the data points as a text file. The only thing I had to remember was to do the three groups separately so I knew which point belonged in which group.

Here’s the fractional anisotropy data replotted to show the distribution for each group. What we can now see is that there is a small subgroup of control kids who have really high FAs. There is also one autistic kid and one kid (arguably two) with SLI who have low FAs. Everyone else is pretty much in the middle.

Verhoeven et al.'s data replotted to show the distribution of fractional anisotropy for each group

On average, kids with SLI have lower than ‘normal’ fractional anisotropy [1], but looking at the spread of scores, you’d be hard pressed to conclude that this was a characteristic of SLI. Likewise, the overlap between the distributions of the autism and SLI groups is almost complete. Hardly evidence for fundamentally different neural mechanisms in the two disorders. 

At the risk of sounding like a broken record, this once again highlights the importance of looking at individual variation within diagnostic groups such as autism and SLI, rather than (or as well as) looking at group averages.

But it also emphasizes a more general point (and this I have to stress is no criticism of the authors of this particular paper).

The data reported in a journal article are really just a snapshot of the actual data recorded, filtered through the authors’ preconceptions about what questions are interesting to ask and how to go about doing that. There’s an imperative to present the data in a neat, sanitized package, with all the rough edges and anomalies smoothed out; to tell a coherent story that will convince reviewers and editors that it’s worthy of publication in a reputable journal. Years of work and terabytes of data may be compressed into just two or three pages.

DataThief only takes us so far. It allows us to extract the information presented visually in the published article, but no further.

Most of the past week has been spent convincing myself that it doesn’t really matter how I analyse my data because the results come out the same regardless. This is reassuring for me, but it doesn’t mean that somebody else, looking at my data with fresh eyes and a different perspective, would not come to an entirely different set of conclusions.

In an ideal world, when a paper is published, researchers should also be able (and encouraged) to publish the data on which the paper is based, as well as the script showing exactly how those data were analysed.

There are, of course, many obstacles in the way and questions to be answered before this becomes standard practice. Who would host and maintain the data? Just how raw should the raw data be? What if the authors are writing multiple papers based on the same data set? Who gets credit for reanalyses of the data set? What happens if a reanalysis shows up an error in the original paper? If the research involves human participants, how do we reassure them that their anonymity will be maintained?

Undoubtedly, there are many more problems that I haven't thought of. But, as scientists, we need to work through these issues and find ways to set our data free.


Footnotes:

[1] The analyses involved an ANOVA with left and right hemisphere as a within-subjects factor. This showed a main effect of group, but no group by hemisphere interaction.


Update:

Originally, I linked to the wrong SFARI article in the third paragraph. That's now fixed. The one I mistakenly linked to reports a conference presentation that does indicate atypical connectivity between language regions in the brains of nonverbal autistic kids (although not the same pathway as examined by Verhoeven et al.)

Further reading:

• Dana Foundation Blog: The Power of Data

Reference:

Verhoeven, J., Rommel, N., Prodi, E., Leemans, A., Zink, I., Vandewalle, E., Noens, I., Wagemans, J., Steyaert, J., Boets, B., Van de Winckel, A., De Cock, P., Lagae, L., & Sunaert, S. (2011). Is There a Common Neuroanatomical Substrate of Language Deficit between Autism Spectrum Disorder and Specific Language Impairment? Cerebral Cortex DOI: 10.1093/cercor/bhr292

Take part in our research on language and auditory processing

We’re looking for kids with autism as well as typically developing kids to take part in our research.

The study is looking at how kids’ brains respond to different sounds, and how this relates to their language and communication skills.

We are using a technique known as magnetoencephalography or MEG for short. MEG works by measuring the tiny magnetic signals naturally emitted by neurons in the brain. It will tell us which parts of the kids’ brains are responding, how quickly, and how sensitive they are to subtle changes in the sounds they are hearing.

It involves absolutely no physical risks. Kids get to go in a “space rocket”, watch a movie of their choice - and get paid!

If you’d like your child to take part, please ring Shu Yau (02 98504314) or email shu.yau@mq.edu.au


Who can take part in the study?

We are currently recruiting children aged 5-12 years, who live in the Sydney area:

• Children on the autism spectrum (i.e., children with a diagnosis of autism, autism spectrum disorder, Asperger syndrome, or PDD-NOS). Our only criterion is that kids need to have at least some spoken language and can complete the different tasks.
• Typically developing children (i.e., non-autistic children with no language or communication difficulties or epilepsy). These children are very important because they provide an objective age-matched comparison. 

What would happen if my child took part in this research? 

Firstly, you and your child would be invited to the KIT-Macquarie Brain Research laboratory to meet the researchers and become familiar with the MEG lab. It is important for us to take time to get to know you and your child before we proceed with the study. We will help the children understand what will be expected of them if they decide to take part. For the younger ones, we also give out prizes and certificates to show that they are qualified MEG astronauts!

If you and your child are happy to proceed, we will start with a short hearing screening test, using headphones, to establish the softest sound your child can hear. Then we will proceed to the MEG, where they will lie on a bed and listen to sounds while watching a DVD of their choice. After the MEG recording, your child will complete some behavioural tasks to give us a record of his/her cognitive, social and communicative skills. These involve playing with toys (for the younger ones), storybooks and computer games.

For some children, a second visit may be scheduled to complete the MEG recording, if they wish, or if the child prefers to finish the behavioural tests on another day.

For parents, we would send you a brief questionnaire concerning your child’s social and communication skills. We’d give you a freepost envelope so you could complete it and post it back to us in your own time, free of charge.

Do we get paid for taking part? 

Yes. We pay $40 for the first MEG visit, and $20 for each subsequent visit to complete behavioural testing.

Where and when would the research take place? 

The study will take place at a time that suits you, and can be split into two or more sessions if needed. The MEG system is at the KIT-Macquarie Brain Research laboratory at 299 Lane Cove Road, close to Macquarie Park station.


View Larger Map  

Are there any risks involved in this research? 

There are absolutely no physical risks involved in the study. If your child became tired or anxious, testing would stop immediately. Unlike other brain imaging techniques, MEG is silent, doesn’t involve things being stuck to the child’s head (except for a swimming cap), and you will be able to stay with your child the entire time. The short hearing test is just a screening test, but we will alert you immediately if we suspect hearing loss/impairment in your child.

What happens to the information recorded? 

The information we record during this study will be treated in strictest confidence and we certainly wouldn’t pass on any information about your child to anyone outside the research project without your written permission. Your child’s scores on the various tests would be coded and stored on a computer with password protection. They would be given an ID number so that nobody outside the research project knows their real name.

How will I find out about the outcomes of the research? 

We will send you a summary of the research project and its outcomes. We will also send you a summary of your child’s scores on the different tests, which you may take to clinicians if you wish.

What happens if I change my mind? 

You are free to withdraw your child from the research study at any time. You don't have to give a reason and you'll still get paid.

Who is conducting the research? 

The study is being conducted by Shu Yau, as part of her PhD, supervised by Dr Jon Brock at the Macquarie Centre for Cognitive Science. It is part of a larger research program funded by the Australian Research Council and Macquarie University.

Would we be asked to take part in other studies? 

If you’d like to get involved in other research projects, we can send you information about future studies. But there is absolutely no obligation for you to take part in these.

I'm still interested. What do I do now? 

If having got this far, you're still interested in your child taking part, please phone Shu Yau (PhD student) at 0298504314 or email shu.yau@mq.edu.au

The ethical aspects of this study have been approved by the Macquarie University Human Research Ethics Committee. If you have any complaints or reservations about any ethical aspect of your participation in this research, you may contact the Committee through the Director, Research Ethics (telephone (02) 9850 7854; email ethics@mq.edu.au). Any complaint you make will be treated in confidence and investigated, and you will be informed of the outcome.

Do kids with autism have big brains?

Like most things in autism research, the idea that people with autism have big brains goes back to an observation in Leo Kanner’s original autism paper, where he noted that some of the kids in his group had larger than normal heads. Over the years, there have been dozens of studies looking directly or indirectly at the issue of brain size in autism. In 2005, Martha Herbert provided a comprehensive review [pdf] of 25 such studies, describing the tendency towards large brains as "the most replicated finding in autism neuroanatomy".

Redcay & Courchesne 2005

Also in 2005, Elizabeth Redcay and Eric Courchesne published a meta-analysis, in which they ingeniously plotted how much bigger or smaller than average the autism brains in different studies were as a function of the mean age of the participants in the study. They concluded that there is an early period of brain ‘overgrowth’, with autistic brains being on average 10% larger than normal. But then growth slows down and typically developing kids eventually catch up.

Redcay & Courchesne's meta analysis. Red circles (my annotation) indicate studies in which brain volume was inferred from head circumference measures.

Redcay and Courchesne's paper has been extremely influential. But their analysis rests on a number of assumptions that are worth highlighting.

First, the data are cross-sectional. Different people are being measured at each of the different ages. This is inevitable because brain imagining technology hasn't been around long enough for a proper longitudinal study be conducted, following individuals over the first decades of their life. But if we think of the curve as being the actual growth trajectory of a person with autism (as Redcay and Courchesne want us to) then we are essentially assuming that a 30-year-old autistic adult in one study is what a 5-year-old in another study will be like a quarter century from now. This is a pretty big assumption.

Second, data for the youngest age groups actually came from measurements of head circumference taken during regular infant check-ups, rather than actual brain scans. Head circumference is correlated with brain size in infants, and realistically it's the only way to study brain size in autism pre-diagnosis. But in order to plot the data on the same graph, Redcay and Courchesne had to make quite a few assumptions about the relationship between head size and brain volume [1].

Courchesne et al 2011

More recently, Courchesne and colleagues published an update, pulling together data from all of their MRI studies. The data were still largely cross-sectional but, this time, they fitted growth curves to the data from autistic and non-autistic individuals.

This gives a clearer sense of the variation within each group, which, even for typically developing children (blue circles), is huge. Having a brain that is 10% bigger than average (as Redcay and Courchesne's analysis suggests) isn't actually all that abnormal. Nonetheless, Courchesne et al.'s curve-fitting led them to again conclude that autism is associated with “early brain overgrowth”.

Individual brain volume data as a function of age (Courchesne et al., 2011). Red squares are boys with autism. Blue circles are typically developing boys.

Comparing the two curves suggests that the difference between autistic and non-autistic brains is largest in the period between around 20 and 32 months (which I've conveniently highlighted). However, if we look at the actual data in this period, ignoring the curves, then we see that none of the autistic kids had brains that were unusually large for their age.

The curve for the typically developing boys is strongly curved because it has to fit the data from the youngest kids (12 to 18 months). The autism data don’t start until around 20 months, so the curve is inevitably less curvy. This gives the impression of a big difference in brain size at around 2 years of age, which I don't think really exists in the data.

That being said, there were quite a few 3- to 4-year-olds who did appear to have relatively large brain volumes.

Nordahl et al 2011

Just before Christmas another MRI study of brain size in autism came out. Christine Nordahl and colleagues focused on a narrow age range, around 3 years, when the "overgrowth" appears most evident.

Before discussing their results, it's worth mentioning their methods: While previous MRI studies have involved sedating the kids with autism in order to keep them still (and get them near the scanner in the first place), Nordahl and colleagues' solution was to scan the kids in the dead of night while they slept. Heroically, they scanned 114 autistic kids and 66 non-autistic control children in this way. I imagine a lot of coffee was drunk.

13 of the 114 autistic children met the criteria for megalencephaly - the technical name for a big brain - defined here as having a brain volume that was more than 2 standard deviations greater than the control group mean. Put another way, 89% of the autistic kids had normal-sized brains. This shouldn't be surprising. Courchesne et al.'s study appears to show something similar, as indeed do most of the previous studies of head and brain size.

Nordahl et al. noted that 11 of the 13 autistic kids with large brains were boys who were reported by their parents as having undergone regression - losing skills that they had previously acquired. In fact, when they looked at all the boys who had regressed, they found significantly larger brains than for typically developing boys. And when they looked back at head circumference measures from the first year of life, the boys who went on to regress had significantly larger heads than the typically developing boys from as early as 6 months [3].

In contrast, increased brain and head size was not found in the autistic boys with no history of regression. Nor was it a characteristic of autistic girls, regardless of whether or not they had regressed.

Distribution of head sizes in Nordahl et al (2011). Green line (added) shows the approximate cut-off for magalencephaly in boys

It's fair to say that Nordahl et al.'s results are still preliminary. As the authors note, they rely on parents' reports of their child's early development, which may not be very accurate. Also, there isn't, as far as I can tell, any precedent for a link between regression and increased brain size [2]. So the results should be treated with more caution than if they had been grounded in previous research findings.

Finally, Nordahl et al. only report total brain volume. They didn't look at what parts of the brain were enlarged or otherwise, or whether different types of brain tissue were affected differentially.

Mechanisms of overgrowth

This brings us to the really interesting question, which is not whether kids with autism have large brains (the answer, if you hadn't gathered by now, is that some do and some don't); but why?

A recent study by Courchesne and colleagues linked increased brain size in autism to an increased number of neurons in prefrontal cortex. But in an earlier MRI study, Courchesne et al (2001) reported that it was the white matter (essentially the axons that connect different brain regions) that was disproportionately enlarged in young autistic children.

In her 2005 review, Martha Herbert speculated that increased brain size might be related to reduced brain connectivity (see also this intriguing paper by Sarah White). It's not entirely clear to me how that would work and I'm not aware of any research showing that larger brains are less well connected. It might be that the link is less direct - perhaps a genetic variation that leads to increased brain size also puts a child at increased risk of autism through a different mechanism.

One thing we can be sure of. Making sense of the association between autism and large brains is going to be far from straightforward. Not only do most people with autism not have large brains, but most people with large brains don't have autism [4]. Large brains have also been reported in relation to other disorders including language impairment and ADHD. And just to add further complication and intrigue, there's evidence that relatives of people with autism tend to have large heads, even if they don't have autism themselves. 

Here, I think, we have autism research in microcosm. A finding that dates back to Kanner, that consistently holds up across studies, but doesn't hold up across individuals within those studies; overlap with other developmental disorders and continuity with non-autistic family members; and theories that don't quite stand up to close scrutiny. It's an illustration of how complicated the answers we're looking for are likely to be, and the fact that the answers are likely to be different for different individuals.

Footnotes:

[1] In order to convert head circumference into brain volume, Redcay and Courchesne adopted the following procedure:

• They took normative data on brain weight at different ages from a Danish post-mortem study and converted this to brain volume based on an estimate of brain density from one of their own studies. 
• They then matched up the brainweights with normative data on head circumferences from an American study.
• These pairs of values were entered into a linear regression analysis, along with data from MRI scans of older children.
• The regression equation was used to convert circumference into volume.

It's difficult to see what else they could have done, but given how influential the study has been, it's important to highlight how many assumptions were involved. It's also a little strange that they assumed (or found) a linear relationship between head circumference and brain volume - when mathematics teaches us to expect a cubic relationship between circumference and volume.

[2] I recently came across a blogpost discussing the fact that none of the kids in Kanner's 1943 study had regressed. I can't track this down (please comment if you know what I'm talking about). But, given that Kanner first noted the children's big heads, it would put him at odds with the authors of this paper.

[3] It's not reported whether the boys with large heads at 6 months were the same boys who had large heads at age 3.

[4] I'm not sure if there is any research on this, but if we define megalencephaly as 2 standard deviations from the mean then roughly 2.5% of the population have megalencephaly (assuming a normal distribution). Even with a generous 1% prevalence for autism, I think it's safe to say that the majority of people with megalencephaly are not autistic.

References:

Courchesne, E., Campbell, K., & Solso, S. (2011). Brain growth across the life span in autism: Age-specific changes in anatomical pathology Brain Research, 1380, 138-145 DOI: 10.1016/j.brainres.2010.09.101

Nordahl, C., Lange, N., Li, D., Barnett, L., Lee, A., Buonocore, M., Simon, T., Rogers, S., Ozonoff, S., & Amaral, D. (2011). Brain enlargement is associated with regression in preschool-age boys with autism spectrum disorders Proceedings of the National Academy of Sciences, 108 (50), 20195-20200 DOI: 10.1073/pnas.1107560108 Download PDF

Redcay, E., & Courchesne, E. (2005). When Is the Brain Enlarged in Autism? A Meta-Analysis of All Brain Size Reports Biological Psychiatry, 58 (1), 1-9 DOI: 10.1016/j.biopsych.2005.03.026 Download PDF