Earlier this week, we posted An Internal Medicine Doctor and His Peers Read the Pfizer Vaccine Study and See Red Flags [Updated]. While most readers responded very positively to the write-up by IM Doc, which included the reactions of the eight other member of his Journal Club, who reviewed the article and its editorial, as they have done regularly with important medical journal articles. We have embedded the Pfizer article from the New England Journal of Medicine (NEJM) below; the link to the editorial is here.

However, some readers took issue with IM Doc noting that two nurses in the UK had suffered anaphylaxis, a severe, potentially life threatening allergic reaction, after getting the Pfizer shot, and criticized the paper and editorial for not having included or added a discussion of any exclusion criteria, since Pfizer’s proxies admitted that severe allergies were an exclusion criterion. From MedicalXpress:

Moncef Slaoui, who is the chief advisor to the US program for COVID vaccine and treatment development, told reporters, “Looking into the data, patients or subjects with severe allergic reaction history have been excluded from the clinical trial.

“I assume—because the FDA will make those decisions—that tomorrow this will be part of the consideration, and as in the UK, the expectation would be that subjects with known severe reactions, (will be asked) to not take the vaccine, until we understand exactly what happened here.”

Slaoui is the co-head of Operation Warp Speed and previously head of GlaxoSmithKline’s vaccine department. Other media outlets and professional medical writers (see here and here for examples) picked up his statement that subjects with severe allergic reactions were excluded.

If you look at the article below, you will see that it is not searchable. That indicates an expectation that it would be read as a print out only. If you put the article through an OCR reader, you will find it make no mention of “exclusion criteria”. Neither does the the separate editorial by NEJM editors. The article does does mention “protocols” in the text, twice, but does not have a link to where to find them, does not have a written URL, nor does it provide a name or location to assist in finding them.

Some critics argued that the protocol (which you need to search through to find the selection process for candidates, including the exclusion criteria, for the Phase III trials) could “easily” be found in the Supplemental Materials and further asserted that any regular reader of medical papers would be able to find then. The fact that IM Doc, who has been reading medical papers for 30 years, and his eight colleagues did not locate them is already significant counter-evidence, particularly since the NEJM’s ad pitch lists the publication’s audience solely as physicians. No doubt scientists read it too, but the eyeballs advertisers want to reach is doctors, not academics or scientists in the employ of competitors.

IM Doc could not find the Supplemental Materials, which in the online version includes an external link to the protocols, because the PDF that the NEJM generates does not include them. Since readers have every right to assume that online and PDF versions of the same article are identical, there was no reason for him to look further.

It turns out that the data waters appear to have been muddied by the NEJM itself. IM Doc and his colleagues found and read the Pfizer paper late last week. It was then on the first page of the site. He sent me his write-up on Saturday the 12th. I went to look at the article and charts on the NEJM site. It was then on the front page of the site. I experienced rendering issues in Firefox but nevertheless was able to look at the article, along with the separate tab in the header area for “Figures/Media”. I also noticed a “Supplemental Appendix,” which I opened. It was a bit of a hodge podge but didn’t contain anything that related to IM Doc’s observations. I did not see any tab with “Supplemental Materials.”

IM Doc did a big revision of his draft on Sunday, which I edited that evening. IM Doc was a bit freaked out Monday AM when the Pfizer article has moved off the NEJM front page, but he quickly located it on the site. I looked quickly, simply registering that I could find it and see the charts, but I did see that the rendering problems were no longer occurring.

I was not the only person who recalls seeing the “Supplemental Appendix” as a stand-alone document with a link to it in the top area of the NEJM site before December 12 (sadly, the Wayback Machine allows publishers to suppress older versions of articles upon request, so there is no image of the article as it first appeared on the first page of the NEJM site).

Fortunately, reader KLG harrumphed in comments about the Supplemental Appendix in comments. KLG is a professor of microbiology and has been doing basic research for 30 years, so he can’t be dismissed as inexperienced in reading scientific papers. I asked for his recollection of what he found when he went to the Pfizer article:

Here is how I remember finding the paper, after seeing the post from IM Doc yesterday. I apologize if this is TMI, but I want to be as clear as possible:

(1) I clicked on the link while in my office and gained direct access through our medical library, downloaded the pdf, and printed it on a high-resolution color printer, which is my standard, old-fashioned practice. I then read through the paper fairly quickly, and thought it was OK/promising but not necessarily complete.

(2) I saw no obvious (printed) link in the pdf to any supplementary materials, so I went back to the online link to the paper through our library (full access to most medical journals). I saw the link to the 12-page “Supplementary Appendix” with the 4 pages of names and downloaded and printed it. This link was on the right side of the screen/page, near the top. This link was not at the bottom of the single page of the paper as I see NEJM.org from home this evening, along with the other links, including Protocols, Disclosure Forms, etc. I am not accessing the journal remotely through our library tonight and would be seeing it as “outsiders” do. The point for me is this: One link to supplementary material should go directly to all supplementary materials. This has been what I have been accustomed to for years. Sometimes the files are ridiculously large and there may be 10-15 of them, but they are all there and easy to find. If one link cannot be managed, then all links should be in the same place on the webpage, listed one after another as S1, S2, S3, etc.

(3) Still, I may well have missed these other links, because while I am very skeptical of “science” direct from Big Pharma, my forensic antenna were not fully deployed until the usual suspects showed up later in the day as I checked back in to see how things were going.

(4) But more importantly, I do not believe for a second that IM Doc and his like-minded journal club (a common mechanism for all biomedical scientists and many clinicians to keep up with current developments) would have missed these materials, if the links were properly displayed as they should have been on day-one. As I mentioned in my previous email, so-called supplementary material has become a thing, for good or ill, in biomedical publishing, and in my experience the links are prominent in strong journals (and NEJM is definitely that, or certainly was when Marcia Angell was editor). Moreover, IM Doc undoubtedly has a subscription to NEJM, which should have displayed the links prominently both in hard copy and online. Based on his every contribution to NC, IM Doc seems uncommonly attuned to both the practice and science of medicine, going back to the beginnings of the HIV/AIDS epidemic, which made a deep impression on all of us who were there, clinicians and non-clinicians alike. That is when we learned to parse the literature and separate the wheat from the chaff. And there was a lot of chaff in the HIV/AIDS literature from ~1983 until triple-therapy was published in 1994 IIRC and AIDS became a chronic, manageable condition for most of those infected with HIV.

In other words, NEJM initially either did not present the Supplemental Materials tab at all or through terrible design, directed reader attention away from it by having a prominent Supplemental Appendix link at the top, which experienced readers would assume contained all of any additional documents, save any others mentioned and/or linked to in the article proper and/or the editorial.

In addition, even after readers in comments had pointed out where to find the exclusion criteria (in the Supplementary Materials, in Protocols, meaning two clicks and a search, when it should have been easier to find), other had difficulty finding it and also deemed the material to be poorly presented:

SE

Thank you for highlighting the exclusion criteria. When I looked at the NEJM article, I was also unable to find them. I noticed that a reader has since posted them, and you updated the post. I also found the on Pfizer’s website. https://www.pfizer.com/science/find-a-trial/nct04368728-0

California Dreaming

My wife is a graduate of Stanford Medical School and is a subspecialist in Internal Medicine. Not infectious disease. She was at the top of her class. She graduated a bit more than 10 years ago. I was so intrigued by this doctor’s post that I wanted to do a little test, admittedly with just one subject. As you can see I was an early commenter and have skin in the game because I live with a doctor.

I wanted to see if she could identify quickly the exclusion criteria just like this doc has now admitted was confusing but actually there. She could not do so quickly – and it took her about 45 minutes to do so and then she stated to me when looking through them how recklesly they were spelled out. She was surprised. Phase 1 stuff all mixed in with the others. She was also very concerned about the fact that there were so many common conditions that had not been studied or included in a wide population impact drug like a vaccine. Concerned about why these are not being more publicized given the fact that we will be trying to vaccinate presumably everyone. Should this not be publicized more with both docs and patients.

Needless to say, this discussion should hopefully put the question of the adequacy (not) of the NEJM presentation of information about the Pfizer vaccine to rest. It was incumbent on the New England Journal of Medicine to provide a clear presentation of information that would enable clinicians to assess a medication intended for the entirely population, yet developed and approved on a corner-cutting basis. They fell short.

Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine – NEJMoa2034577