Editor's choice

Doctors' information: excessive, crummy, and bent

Al Gore, vice president of the United States, has described the mess better than anybody. "Our current information policy," he said, "resembles the worst aspects of our old agricultural policy, which left grain rotting in thousands of storage files while people were starving. We have warehouses of unused information 'rotting' while critical questions are left unanswered and critical problems are left unresolved." We have much too much information of poor quality, too little that's good, and no effective way of sorting it out.

All this matters because most of what doctors do is of small benefit and potentially harmful. It's thus easy to confuse benefit and harm. When effects are small, large trials are needed to rule out the play of chance. But size cannot compensate for bias. Big numbers can mislead if data are biased—because investigators are keener on one outcome or because negative results are buried (p 640) and positive ones duplicated (p 635). Randomisation is needed to avoid bias.

Tribes of meta-analysts are busy trying to clear up the Augean stables of medical information. As Trish Greenhalgh describes (p 672), they pose clear and answerable questions, search for all relevant trials, discard those that don't meet specified quality criteria, and then draw conclusions from the relevant studies—perhaps by statistical amalgamation. They are much criticised for their pains, and many doctors remain sceptical of meta-analysis: "Put together lots of small rubbish studies and you have a big pile of rubbish studies, not an answer."

A collection of papers in this issue—summarised by David Naylor in an editorial (p 617)—look at several aspects of meta-analysis. Critics of meta-analyses ask whether they are good or bad. As our cover (explained on the contents page) suggests, the answer, as for any methodology, is that they are both. Editors and readers need to become more adept at telling good from bad. We must be sceptical about those that are put together from many small studies. We should question those that don't include an attempt to locate unpublished studies—because such studies are more likely than published ones to have negative results. (Our amnesty for unpublished trials (p 622) might help.) We should look for a "funnel plot," which will give us some help in determining whether there is likely to be bias in the studies included in the meta-analysis (p 629).

Finally, we should remember that the information that is drowning us is biased. Whatever technique we use to try to reach answers from the information— no matter whether it's a systematic review or grabbing the closest paper in the library—we cannot avoid that bias.