A Fools Gold Standard

Austin Frakt at The Incidental Economist has a review on Angus Deaton's critique of randomized control trials as the "gold standard" of science.  Frakt suggests that one major advantage of RCTs is that they are "conceptually simple" requiring less mathematical or statistical training in order to understand the results.

Ideal randomized control trials provide two pieces of information:

1.  An unbiased estimate of the average treatment effect, and

2.  An unbiased estimate of the minimum proportion of the population who benefit from the treatment over its tested alternative.

I argue here that learning the average treatment effect is not terribly useful.  The question is whether we can learn that there exist people who benefit from the treatment from settings as conceptually simple as RCTs.  Here I suggest two.

The first is the case where a treatment becomes available at a certain point in time and we can look at what happened before and after.  The chart below shows the survival rate of HIV-infected patients before and after the introduction of various drugs that became the AIDS cocktail or HAART regime.  This chart was published in the New England Journal of Medicine in 1998 and was one of the first pieces of evidence that HAART was enormously effective in reducing deaths from AIDS.

The second setting is perhaps more controversial.  It is the case where we are willing to assume that the people in our study are selected into the treatments that are generally going to make them better off.  Technically, we need people to be selected into the treatment that "first order stochastically dominates."  Some writers call this assumption the minimal requirement for rational decision making (Hadar and Russell, 1969).

The chart below is from the BLS and reports the familiar result that people who attend college earn more money.  If we are willing to assume that people choose the education level that is more likely to give them the higher earnings then the result presented in the chart shows that for at least some people a college education increases wages.

But haven't we been told over and over again that this chart suffers from "selection bias"?  People who attend college may simply be those who would have earned more money anyway.  If college doesn't do anything to people's future income then these people are certainly spending a lot of money to play beer pong and attend college football games.  If they are spending money for nothing, then claims that college attendees are smarter than the average Joe are pretty suspect.

To be clear the claim is that college increases incomes for some people not necessarily for all people or even many people.  The chart and the assumption that college students are not idiots suggest that college has a causal effect on income.  No RCT needed.