By Gerrit Van Wyk.
Lies, damn lies, and statistics.
When scientists hold differing beliefs or opinions, they attack the other party’s theory, facts, or method, and within that lies a conundrum.
The Cochrane Library is a database of systematic reviews and meta-analyses of well-conducted controlled trials for providing guidelines about practicing medicine based on evidence. It recently published a report about what evidence there is masks prevent the spread of Covid-19. The lead author summarized the result in an interview, during which he said there is no evidence masks, even N-95 masks, make a difference, which triggered an immediate backlash.
The media, as it does, printed articles both applauding and condemning the results, adding to the confusion, and masking war. For example, the New York Times article is titled “The Mask Mandates Did Nothing. Will Any Lessons Be Learned?”, as opposed to The Washington Post’s “Yet another study on masking causes confusion”, which is not helpful at all.
One observer quoted in Vox called the lead author “eccentric”, and then proceeds to say the panel asked the wrong question, attacks the method, questions whether the data was interpreted correctly, calls the report scientifically irresponsible, and ends by saying we need more studies to settle the question. He rightly points out to the fact that scientists when studying complex phenomena are obliged to make judgment calls, and therein lies the rub.
When you introduce humans into research, you increase the complexity of the research immensely. Most researchers treat this component as a black box which means the subjects in general behave predictably and act the same way. My own experience of research is nothing can be further from the truth. Just like the Hawthorne experiments in management, in medical studies people try to give researchers what they think they want, rather than truthfully reporting on what really happened.
For example, in Covid-19 mask studies, do we know what kind of mask people wear, did they wear it all the time, and did they wear it correctly. We all remember people wearing anything from rags to N-95 masks bought on the Internet, those with masks hanging below their noses, and those constantly slipping them on and off during the pandemic. Knowing all this matters a lot.
A second problem is doing randomized double-blinded controlled studies during the pandemic would be highly unethical, so superimposing self-reports about masking and non-masking on top of that muddies the water even more.
Thirdly, no paper reports on the amount of mask use indoors, outdoors, whether it was worn amongst people or not, if there were, how many people, and have any idea if there were people, how many were infected. If I wear a mask and there is no-one else, or wear it and no-one around me is infected, there is a 100% the mask will prevent infection, which tells us nothing. This matters a lot. A tongue-in-the-cheek paper reported on a randomized double-blind study in which half the participants were given parachutes and the other half not. They were then asked to jump from an airplane and found 100% survived without injury, which leads to the conclusion parachutes are not necessary from jumping from an airplane. The study, following strict academic protocol, didn’t state the airplane was parked on the tarmac. You cannot just assume conditions are all the same or somehow evens out.
Cochrane reports and most medical research today is based on statistically manipulating or trawling datasets, and as Ioannides pointed out, most are statistically flawed. It is a dirty secret we don’t talk about, the reasons for which is part of human complexity. Also, combining studies for a meta-analysis depends on peer reviewed published papers, and for complex social reasons, only a small number of submitted papers are published, and a large amount of research is not submitted, which creates the file drawer problem; most research and data remain hidden from view, which is important.
If you sit in a laboratory and observe what goes on from a social perspective, it quickly becomes clear judgment calls are indeed made, but also that these calls don’t follow a logical protocol or sequence, but more likely involves power dynamics and office politics, which significantly impacts on outcomes. I once asked a statistician for help with data analysis, and before he looked at the data asked, what do you want to prove? That is the human dynamic. There is ample evidence panels who perform meta-analyses for creating reports and protocols suffer from the same biases and dynamics.
All this means the criticism of the masking report is very valid, but not for the scientific-academic reasons given; the real problem is hidden in the complexity of human social dynamics.
It means the for and against masking arguments will not be solved anytime soon, and likely never will be, and people in general and scientists particularly will continue espousing their own favorite beliefs and biases. It is doubtful the question can be solved scientifically.
My personal approach to data is to follow the pragmatic tradition; if the facts available to me prove to be useful, they are worth using. It means if you choose to wear a mask, it must be of the right type, must fit properly, and must be used all the time. It is of little use if no-one around you has an infection, or outdoors, or if you allow infected people to be within around 2 meters of you (which is somewhat controversial as well). Under these conditions, mandating mask use creates a regulatory nightmare and all you end up with is resistance, many masks of the wrong material, used wrongly, or sporadically, as we all saw during the pandemic, which is counterproductive.
In the end, science and politics are complex hidden social games to which we should pay attention. If we choose not to, the flaming and shaming on Twitter, Facebook, in the media, in parliament, and elsewhere will continue without end, which pragmatically serves little purpose, other than the psychological payoff. If you want to be critical of Cochrane and other reports, it would be better doing it for the right reasons if we want to move forward.