GS5002: Academic Professional Skills and Techniques, "Journal Club on Big Data and a Bewildered Lay Analyst", 2021 (AY21/22 Sem 1)

Session #3, 27 August 2021: Vul et al. "Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition"

The paper by Vul et al. discusses a major statistical error which was very common in early fMRI studies. This type of error is usually referred as ‘voodoo correlation’ or ‘non-independence error’. After numerous papers and press exposing this issue, it became a huge blow on the faith of general public towards fMRI studies. Now, fMRI studies have become more cautious and the problem of ‘voodoo correlation’ is by no means confined to fMRI studies.

Organization of the session, lead by Zhang Shaoshi

Part I, Background information:

This part deals with background knowledge of cancer biology and statistics that is needed to understand the paper. Some keywords are highlighted below to look up background literature, Wikipedia, etc.

• What is fMRI? How the fMRI data are processed?
• Brain-behavior association
• Pearson correlation, reliability

Part II, The paper by Vul et al.

This part presents the paper itself. We want to know the key technical details and the key messages.

• How do the authors derive the hypothetical upper bound of the effect size of brain-behavior association?
• What is non-independence error and how does it lead to inflated effect size?
• What are the suggestions made by the authors to address this error?

Part III, Possible points for discussion

This part discusses the Vul et al. paper, hopefully in depth. We want to know whether there is any methodological issue, any doubt on the conclusions/key messages, any suggestion for improving the paper. Some pointers for discussion include:

• Are the problem and solutions suggested by the authors valid?
• Other possible solutions to the ‘non-independence error’?
• What are the common statistical errors in your field?

Instructions

The journal club has 4 sessions. We will discuss only 1 paper in each session. I will pick the paper for the 1st session, to set the scene. Hopefully, you will suggest the papers for the subsequent 3 sessions (we will choose by a simple vote from among the suitable papers you suggest.) Any paper can be suggested, so long as it (i) concerns data analysis (esp. big data) and (ii) contains “controversial” analysis or methodological issues that you think are worth for your classmates to appreciate.

For each paper, the presentation is divided into 3 parts: (i) background of the topic/paper – to help students who lacks domain knowledge, (ii) the paper itself - focusing on technical details, and (iii) discussion on the paper. And you will be divided into 3 teams, each team presents one part. The team will rotate through the 3 roles over the 4 sessions. This also mean everyone has to read every paper (plus some related papers/webpages which are helpful for understanding the paper being discussed.)

The grading will be based on presentation (50%), asking and answering questions during the discussion (50%).

Wong Limsoon
27 Aug 2021