The Zumbo (hypothesis) Test

Here in Australia, we are at the tail end of a reality cooking competition called "Zumbo's Just Desserts". In the show, a group of hopefuls compete in challenges where they produce desserts, hosted by patissier Adriano Zumbo. There are two types of challenges. In the "Sweet Sensations" challenge, they have to create a dessert from scratch that matches a criterion such as "gravity-defying", "showcasing one colour" or "based on an Arnott's biscuit". The two lowest-scoring desserts from the Sweet Sensations challenge have to complete the second challenge, called the "Zumbo Test". In this test, Zumbo reveals a dessert he has designed and the two contestants try to recreate it. Whoever does the worst job is eliminated.

I find it very interesting that the Zumbo test is the harder of the two tests. In the Sweet Sensations challenge, the contestants can choose to use whatever skills they are already good at, and design their dessert in a way that they can personally achieve. In the Zumbo Test, the contestants have no control over the techniques that are required, and must try to do things they are not familiar with in ways they may not have seen before.

And why am I talking about this? Because my medical students find themselves in similar situations. Our medical students have two projects to do as part of their research curriculum during their third year. One project is a research proposal: they work in a group with a supervisor to plan a hypothetical research project, including ethics, literature review and (this is where I come in) statistics. The other project is a critical appraisal: they work in pairs to analyse a published article, including where it fits in the research, the writing, the importance and (again where I come in) whether the statistics is appropriate.

I have noticed over the years that in terms of statistics, the critical appraisal is harder than the research proposal. A meeting with students about the critical appraisal usually takes twice as long as one for the research proposal, and twice as much preparation for me. Many more students come to me to talk about the critical appraisal, and the ones who do come are more worried about the statistics they find in the critical appraisal than the statistics they need in the research proposal. Why is this?

When watching Zumbo's Just Desserts, it occurred to me that the reason why is the same as the reason the Zumbo Test is harder than the Sweet Sensations challenge.

When doing your own research you can choose to only investigate questions in such a way to use the statistical methods that you understand. Even if you need a new statistical method, you just need to learn that one. Either way, you have complete control over your own decisions and know the things you are measuring and what they mean. It's just like in the Sweet Sensations challenge the contestants get to make all the choices and use methods they are familiar with.

On the other hand, when reading someone else's research, you have no control over the wacky statistical methods they choose to use. Even if they are the appropriate ones (they often are in medicine, actually), the paper almost never describes how the researchers decided to use them – it just says what they used. And they often measure new things in new ways that you don't deeply understand. It's just like in the Zumbo Test the contestants have to do things that are new to them in ways that are new to them.

It's much much harder to understand the statistics in someone else's research than it is to make your own.

Let's just hope we don't eliminate all the students by asking them to do it with less support.