Forest plots

A couple of weeks ago I sat in a SAS course to refresh my memory about graphics in SAS. Near the end of the talk the question was posed if SAS could do forest plots. The speaker, not knowing the term, asked for an explanation.

A forest plot is an arrangement of bars which represent clinical trials or other studies roughly addressing the same research question. The ends of the bars represent the (95 per cent, of course) confidence limits associated with the estimated effect. At the point estimate a square is drawn. Its area represents the weight given to the respective study in the elicitation of a common effect estimate as part of a meta analysis. The common effect estimate is represented by a diamond whose width corresponds to the confidence interval of the pooled estimate.

I had to admit to myself (and, as of now, to you) that I was unaware of the term forest plot too. I am well aware of the funnel plot, where the bars are horizontal and arranged on the y axis according to the number of observations in the study, or the standard error (in which case the y axis points downwards).

funnelplot.png

The idea is to get an idea of publication bias. If anything like that exists, weak studies with few patients are more prone to fall prey to it, leading to an asymmetric picture, looking unlike an upturned funnel. My first reaction was "a rose by any other name" and that a forest plot does not really convey anything new. Both funnel and forest plots stem from the Evidence Based Medicine pundits. The funnel plot seems to be a bit older; the first Medline search result is from 1988. The first search results for "forest plot" in Medline carry titles like "Biological screening of rain forest plot trees from Palawan Island"; I haven't found a citation relevant to this subject from the last century at all.

Maybe the choice of the descriptive tool is a way of forgoing the conclusion: The main task of a funnel plot seems to be to expose rather to investigate some publication bias. If your goal is to distill a common estimate from different results, you'd better not undermine your results with a tool designed for that goal. Rather, you will use the forest plot as a more neutral means. But maybe I am being cynical here.

First of all, both the forest plot and the funnel plot reveal in two graphical features what is essentially the same feature of the underlying study. The weight of the study is, of course, deeply linked to the width of the confidence interval. This holds ex forteriori for the standard error; yet also to the sample size. Therefore, some versions of the funnel plot only plot the point estimates as dots, hiding nothing. Yet the urge in properly educated statisticians is strong to pay the confidence interval due respect and never only plot the point estimate. Giving the confidence interval implicitly by some position on the y axis just doesn't cut it.

Once the confidence intervals are part of the plot, another issue arises. The weaker trials are rendered by wider bars "and, perversely, were the most noticeable on the plots". Therefore, in the desire to appeal to intuition rather than to sense of abstraction, some way of stressing the short bars had to be found. Enter squares.

Anybody who has done plots where a measure is represented by the size of a figure knows the problem, as does anybody who has ever populated an aquarium without putting too much thought into foraging behaviour: The larger objects will swallow the smaller ones. In this case, the problem is virtually by design: The shortest bars will be covered by the biggest squares. If you choose the squares to be small enough you might have another problem, especially with studies of varying size: The smaller objects will not be larger than the hairline width of the bar so you will have to find a way to make the location of the point estimate noticeable. Some other problem is that the squares might overlap other bars, or even each other. Somebody came up with the brilliant idea (PDF) to replace squares by rectangles of fixed width. Now we get bar-swallowing blotches that grow linearly with size of the study, and not just the square root. To give the Evidence Based Medicine community credit: This variant didn't gain popularity.

Once the squares were introduced, it wasn't felt necessary to stress the, albeit important, feature of the size of a study by three graphical features. Exit the y axis. Enter equidistant bars.

One ambiguity in forest plots is whether to draw the bars vertically (PDF) or horizontally. The vertical arrangement gives some justification to the name. The horizontal version reminds me (and, I presume, my cohort-mates) of a scene of the arcade game Frogger where the logs are floating in a river and used as stepping stones. Indeed, it should have been called Frogger plot to avoid connotations with completely unrelated methods (and unrelated Medline search results to boot, see above). Stuff SEGA and their trademark rights, this is freedom of research. As legend has it, they can't even agree on the spelling of for(r)est plots. As in 1990's grrls, the number of r's is optional. The horizontal arrangement has the immense bonus that single studies can be labelled. And, remember, all you Tufte-consuming statisticians, always value labels over legends and footnotes.

Now we have grudgingly accepted that both square size and confidence interval bars are part of the plot, and that the bars are arranged in an equidistant fashion, we gain one more degree of freedom to convey information, that is the order of the bars. It should not be wasted by ordering them arbitrarily. So what should the order convey?

You can order by size of the trial, in which case you re-introduce the funnel plot in a distribution-free version. But we already told the reader by two features how precise the trials are, thank you. A more sensible possibility is to order by point estimate, as done in another SAS macro (PDF) (I hope I have helped answering the initial question).

forestplot-sas.png

I like this version; it gives you immediately the idea of dependence between precision and point estimate, what the funnel plot is intended to show. Then again, this arrangement does not add any new information but only stresses what is already said. More convincing to me would be the order of publication dates.

The statistical approach to meta-analysis is essentially to treat a single study as a draw from a pool of independent objects. Anyone who has ever undergone the painstaking process of a study, be it a multicentre clinical trial or a case-control study in quest for a cause of a disease, will have a hard time to accept it being treated as just one of some interchangeable objects drawn from an urn. Obviously the studies are not independent as they (hopefully) cite each other. A new study about a research question addressed by a previous study is conducted for a reason. Take my words with the contents of a salt cellar as I am speaking as an outsider here, never having conducted a meta-analysis. But it is precisely for that reason: Although the aim to aggregate a common estimate is laudable (she who claims you couldn't do it because every study is unique questions external validity of any study), I feel that the statistician's role is one of the less important in the endeavour.

Therefore, any information which sets the single study apart should be conveyed in the plot. Labelling the bar with author and year, but also with peculiarities of the study among the others (slightly divergent hypothesis, choice of collective …) adds crucial information to the graph. As in Frogger, it is important to notice the alligators among the floating logs — the studies you should rather cautiously use as stepping stones. If space allows, one may even plot the mutual citations between the study publications as a (hopefully acyclic) directed graph next to the forest plot.

To sum it up, there are some upsides of forest plots over funnel plots. Funnel plots have an advantage where there are many studies which are better represented by just one dot. If there are fewer studies, forest plot them, but take notice of the individual trial as much as space allows.

Add a New Comment
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License