The Decameron of poor research

First tale

Federigo, a research fellow, pleads for help to defend Vancouver; yet, after a secret serious discussion on the deeper philosophical meaning of scientific authorship, he spontaneously applauds and kneels in admiration before his Master

Aldobrandino was collaborating with Federigo, a young physician trained in world class institutions. After lots of work, tons of analyses, and several revisions, Aldobrandino sent the final manuscript to Federigo for his approval. In response, he got a confidential email. “Dear friend,” Federigo wrote, “this has been a wonderful collaboration. However, I have bad news. There are seven authors now. Messer Guglielmo, my director, is eating my flesh, but I don't want my career to end. Monna Nonna, my wife, needs papers to get a job. Peronella is directing the lab next door—we must be kind to our neighbours and build collaborations. Tedaldo, my young fellow, also must build his curriculum vitae. Lastly, Messer Guglielmo had another co-author to add. He forgot his name, but he will let me know tomorrow.” A postscript followed: “Have you seen my work at XXX? Many consider XXX the top university worldwide. In my best paper, of seven authors, three contributed nothing, and one had no clue of the project's existence until he received a reprint.”

Scientifically speaking, Aldobrandino went bananas. Federigo apologised and pleaded for help to face the bloodsucking Messer Guglielmo. He promised that, with Aldobrandino at his side, he would stand firmly against his director in defence of the Vancouver authorship criteria. In this spirit, a meeting was arranged where Alobrandino and Federigo met with Messer Guglielmo two days later. Federigo asked to talk first. Making a request to talk when your director is present is indeed a valiant act, and Aldobrandino was pleased that his young colleague seemed so determined. However, instead of praising Vancouver, Federigo started praising his worthy director. He made a deep and moving confession, asked for forgiveness, and urged that his Master should be the senior author and sole arbitrator of authorship.

Albobrandino understood that something must have happened the previous day that had changed the stance of his colleague; he asked his name to be removed from the manuscript and thanked them both. Much later the paper was published without his name.

Second tale

A completed non-randomised study by Chichibio and colleagues is rejected, but then it miraculously becomes randomised and gets published with full honours

Chichibio submitted to a prestigious journal the manuscript he had written with countless others, as in the previous tale. One reviewer noticed that, although the title and abstract promised that the study was randomised, the methods section clearly revealed the contrary. This reviewer mentioned the discrepancy and later saw that this comment had, in fact, made it to the authors.

The manuscript was rejected, but it was published eventually in another journal. Just as Rodney Danger-field swore that he would fix it so that his wife would never be on the couch with another man again, and then sold the couch, so did Chichibio and colleagues fix the problem. The word “randomised” still appeared prominently in the title and abstract. However, the authors were sufficiently considerate to readers to relieve them of the unnecessary trouble of noting that this was not a randomised trial. The contradictory wording from the methods had been appropriately erased. As for the discussion section, this also gracefully expounded the merits of this unique randomised clinical experiment. The medical literature was being enriched with yet another superb study without a section on limitations.

Third tale

Efigenia, a young physician in search of a faculty job, is forced by seasoned professors to repent the mortal sin of having been involved in evidence based medicine and considers becoming a nun instead

Well, I doubt randomised trials mean much to many physicians worldwide. With solid clinical training and several publications in prestigious journals, Efigenia sought a junior clinical faculty job.

The interviewing Professor lamented the fact that Efigenia was “working on controversial topics, randomised trials and even meta-analyses” and gave seasoned advice to the youngster: “Gone is your reputation as a physician.” Efigenia felt her interest in evidence based medicine was worse than filthy extramarital affairs. Well, even if Efigenia were to repent (an act which she certainly should do, at a minimum), the damage was probably irreversible. “It is so sad, you won't ever be able to think straight again after this exposure,” claimed the Professor, nodding his head. Nevertheless, it all ended in a friendly tone: “I am terribly sorry, we simply don't want mathematicians here.”

Notwithstanding the first shocking disappointment, the second interview went better. “Seat please,” the Associate Professor said. “I looked at your CV.”

“This is good news,” Efigenia thought.

Portraiture of sublime research: lofty academics gracefully beguiled by idealistic sponsors (The Decameron (1916) by John William Waterhouse, held at the Lady Lever Art Gallery, Port Sunlight, Merseyside)

Credit: BRIDGEMAN ART LIBRARY

“So, I understand you have worked on, aaah…”the Associate Professor stopped trying to find the word. “Aaah, what's the name, aaah, yes, β analysis.”

“Yes, of course,” Efigenia replied, thinking regretfully, that she had worked and wasted her life on α trials, β analysis, and other types of γ particles.

The next she expected to hear was the Hamlet motto, “Get thee to a nunnery,” but let us stop here to avoid mixing Shakespearean and Italian literature.

Fourth tale

Beltramo, a regulatory agency statistician, in vain tries to prove his points; at the end, bad research is justly rewarded by regulatory approval and publication in a prestigious journal

Who's afraid of evidence based medicine? At a regulatory meeting to discuss the approval of a drug, Beltramo, the regulatory agency statistician, pointed out that the singular pivotal randomised trial was badly flawed. Masking was forfeited, explained Beltramo. “Come on, we have seen worse,” was the reply he got.

Allocation concealment was subverted, Beltramo continued, but the administrators asked for a dictionary: “Allocation concealment? What is this?”

Frontispiece from a 17th century English translation of The Decameron (private collection)

Several patients were randomised from the wrong stratum. Practically, the investigators had chosen at their will the patients “randomised” in each arm. What did the agency have to say for this? “We are late for lunch,” the chief officer replied.

Eventually, showing its well founded and well funded respect for bad research, the regulatory agency approved the drug that was anxiously anticipated to make a big splash in the market. Only utopians would argue nowadays that the money is not more important than the science. However, a crust of scientific glory is not bad either. It is thus not surprising that a few months later, the trial found its way into the pages of a prestigious journal. The journal actually seemed to have done an excellent job in silencing the flaws of the study.

Fifth tale

Despite the reluctance of Cimone, the methodologist, a team of top researchers is rushing heroically to meet the abstract submission deadline of an international meeting—at all costs

All trials have their problems. Data were still being collected when the principal investigator telephoned Cimone, the methodologist. The deadline for the international conference was imminent. Some centres had sent no data, others were uncertain whether they had even participated in the study, but then “Who cares about the data, just do it,” cried the principal investigator on the phone. Cimone declined to meet these high standards.

Next morning, Lazzarino, a diligent clinician working with the principal investigator, showed Cimone the analyses he himself had run overnight. Cimone took a look. It soon became clear to him that all these analyses shared something in common: they were all wrong. After abusing all principles of statistics, Lazzarino had made the drug seem effective, despite it being completely ineffective. In vain Cimone tried to explain. Lazzarino remained unmoved. Cimone finally exploded, “We cannot mislead others and ourselves so blatantly.”

Lazzarino responded quietly, “OK, let me ask. Have any others ever made such statistical mistakes in published papers, in good journals?”

Cimone replied without second thought, “Sure, these are common major mistakes.”

Lazzarino jumped from his seat: “If others have made these errors, why not us? We are fine, it's on its way.” He thanked Cimone kindly. “Sorry, I must run, the abstract is due by 5 pm.” The presentation made a great impression at the international conference.

Sixth tale

Alessandro, a statistician, fails to massage appropriately his mistress (the data) and insists on disseminating (rather than inseminating) the results of a study that would be unfavourable to the sponsor; there is no happy ending here

Contrary to the previous story, Alessandro did perform an analysis of his data in exact accordance with what the study protocol had specified. The protocol was well thought, the study was well conducted, everything was in place. Analysing the data of a well designed and well conducted study is a superb, if uncommon, experience nowadays. Well, this analysis showed that the product developed by his employer was not, in fact, superior to the competitor, as had been hoped. Still, Alessandro reasoned, the facts are the facts.

This position turned out to be unpopular. The Director of Alessandro's department told him to challenge this result. When Alessandro asked how to challenge it, he was told, with some exacerbation on the part of the Director (a more seasoned employee would have known, without having to ask), to play around until a model is found that proves “what we all know to be true.” Alessandro had a very short tenure with this particular institution.

Seventh tale

Guiscardo wastes his chance of a lifetime to become a star of the medical-industrial complex; it is only fair that he then disappears in vulgar anonymity

What we should take as the key message from the previous story is that one must respect the powerful. Guiscardo was a young, hard working physician-scientist. He was pleased and flattered when Professor Tancred called. The school was organising a research conference, and Guiscardo was invited to give a plenary. Guiscardo proposed as a topic, “Medical errors and adverse effects of medical treatment.”

After some silence, Tancred promised to “discuss this with the other organisers.” Several days later, he called back. A different topic would really be preferable. Guiscardo was pitifully stubborn.

Giovanni Boccaccio. From the Villa Carducci series (c 1450) by Andrea del Castagno, held at the Uffizi, Florence, Italy

Credit: UFFIZI, FLORENCE/BRIDGEMAN ART LIBRARY

Then an even more powerful Professor called. The organising committee was unanimous. Influential journalists were expected; discussing medical errors was crazy in front of journalists—or even in front of physicians to be honest. More importantly, the school hoped to strengthen ties with the industry; side effects were a nuisance. Here was the deal. Guiscardo had also worked on infectious diseases; he could talk on “Introducing new antibiotics in the community” or anything on new drugs of his liking, a prime opportunity to show off. Guiscardo apologised for being unable to contribute to the conference.

The meeting was a great success. Many journalists and drug reps attended, and even physicians were present. As for Guiscardo, he never again received a similar invitation and remained at best a totally obscure and controversial figure among his local peers.

Eighth tale

Having trashed Tedaldo, who is pleading for a sound study design, the sponsor of a study shows his open mind and inquisitive spirit in considering plans for an alternative analysis

Unavoidably, research needs money and generates money. Tedaldo was involved in a meeting to discuss the possibility of using public money to fund a clinical study that would use a trick to ensure that it would get the politically correct result. Tedaldo disagreed about the plan, but he could only become annoying to the group. He then suggested a more appropriate approach to the study design and analysis.

The sponsor insisted on the original plan, which would be more likely to yield the desirable conclusion. However, he was willing to consider also the analysis that Tedaldo was proposing, if the need were to arise unexpectedly. After all, he admitted after the study had been funded, sometimes one gets better results with methods that originally don't seem so friendly to the marketing needs. One needs to be open minded, and research always rewards the inquisitive spirit.

Ninth tale

Despite the annoying intervention of young Landolfo, the members of the board of directors of a prestigious agency tactfully and transparently award a juicy grant to themselves

Landolfo was a board member in another public health government agency. Weirdly enough, he was young, unknown to the media, politically unconnected, and even had a research record. These major deficiencies clearly insulted the lofty non-meritocratic tradition of the agency. The presented proposal was one page long, showing the requested funding (a long number indeed), and a list of 20 investigators to be funded. These 20 investigators included only members of the board itself and their close buddies and academic subordinates.

Carlo, a powerful lobbyist, was not expected at the board's meeting, but unfortunately he showed up. Paganino, the proposing investigator, prudently and quickly added in pencil the name of Carlo to the proposal. He then apologised to this “highly respected scientist” for this “unfortunate omission” that was being properly corrected on the spot. The correction made, everybody praised this “extremely well developed, worthy project of great national importance.”

However, Landolfo argued that it is unfair to spend fortunes without any competition. He also requested, as a minimum, to see a protocol for the proposal. Paganino apologised that Landolfo was missing from the funding list but offered to correct this second “unfortunate omission” immediately, pencil at hand. Landolfo declined the offer and asked for a vote. The vote naturally turned out 8 to 1, with the money awarded without any request to see any protocol ever. Shortly thereafter, ungrateful Landolfo was deservedly relieved of his duties.

Tenth tale

Saladin, a superb academic leader, reflects on his successful career and on the difficult paths that lead laborious scientists to excellence; then he proceeds with his daily agenda to save the world

It all boils down to running the (academic) show. That morning Saladin reflected on his very successful career and how he had nurtured the leadership of tomorrow. While other academic leaders had made professors among sons, nephews, and pet dogs, Saladin just gave away PhDs to his protégés, poor kids rewarded for fetching and carrying diligently for him.

On one occasion, Gabriotto, an examiner, realised that the PhD candidate had performed practically no analyses at all. “Why, don't you know?” he was told, “This is ultra-modern research that overcomes the need for any analysis.” Gabriotto apologised for his ignorance. But then, everybody knew that the candidate had done nothing until two weeks ago—yet here was a completed thesis. Where did this come from? Gabriotto asked for the data records. “Records?” Saladin smiled, “Let's have a beer, my friend. Trust me, we have a big heart, who needs records?” Gabriotto also realised that the one piece of any worth in the thesis had actually been defended recently by another fellow. Whose work was it? “Take it easy, my friend,” Saladin replied, “We are all one big family.”

Gabriotto didn't feel like family. Yet, a touch of gentle family pressure (simply Gabriotto's private and public annihilation, no more than that) helped approve the thesis. Saladin regretted that Gabriotto could not understand that fetching and carrying for your Master makes for excellent scientific training. His skilled protégés would always get well paid jobs in circles where the virtue of servility is still highly valued.

Other academic leaders had relied on religious faith. In the “martyrdom tenure track,” when Saladin was young, a professorship cost roughly 20 000 bent knees. The aspiring young faculty had an excellent chance of promotion after making 40 bent knees per mass in 500 masses, making sure he is always in front of the eyes of the pious members of the promotion committee. Yet faith has deteriorated since, so current junior faculty members should not dare think of promotion without a minimum of 100 000 bent knees to their superiors. Still others had exploited their solid devotion to specific political parties, nightclubs, football teams, or combinations thereof.

Pitiful old style. Saladin could adjust his beliefs as requested. He befriended everyman, to undermine everybody blocking his way. His power was money and dexterity to avoid competition unless he knew the winner before the call for proposals. Only idiots apply for grants with 10% acceptance rate when funds are a telephone call away. Telephones beat super accelerators as research tools. Well, enough of that reverie, time to dial the first number for the day.

Let us leave this academic leader do his serious business: the edification, progress, and overall advancement of mankind rest on his touch pad. And let us end these sad stories here, noble ladies and gentlemen. Forgive our insolence, “if in reading any of these tales you find any pleasure.”