Can Observational Data Measure Drug Effectiveness?

Survival and drug use among AIDS patients on
California Medicaid.

It is not often that a graph presented in an economics paper makes me tear up.  But the one to the right always does. Economists, Mark Duggan and William Evans, present this graph in their paper on the effectiveness of drug treatments for AIDS.  The graph shows the change in quarterly mortality for California AIDS patients before and after the introduction of the AIDS cocktail (HAART) in 1996.  The graph shows that quarterly mortality drops from 8% to 2% in the four years around the introduction of HAART.  The graph is based on claims data in California's Medicaid program.

Often in science we are interested in measuring the causal effect of some proposed treatment or policy.  For example, we may be interested in whether HAART causes AIDS patients to live longer.  That is, if we give a particular patient HAART would that patient live longer than they would have lived if they had not received the treatment.  In his book, Causality, Judea Pearl argues that to test whether or not a casual relationship exists we need to conduct an "experiment" where the researcher has the ability to adjust one variable and observe what happens to the variable of interest.  If we want to test the causal relationship between HAART and survival, Pearl says we should conduct an experiment where the researcher is able to control whether or not a patient receives HAART.

Some economists argue that the experiment does not have to be a randomized control trial, rather it could be a natural experiment.  A natural experiment is like a randomized control in that some sub-group of the population of interest receives one treatment, while a different sub-group receives a different treatment.  Importantly, all people in each sub-group have no choice about which treatment they receive.  Moreover, the two sub-groups are otherwise similar (except for the treatment they receive).  The difference is that the experiment occurs naturally in the world.  That is, there is something that happens in the world or some characteristic of the world that leads to different treatment for different sub-groups.

In Duggan and Evans (2008), that characteristic is time.  Prior to January 1996, HAART did not really exist.  It wasn't a treatment option unless the AIDS patient happened to be on AZT and in a clinical trial for Epivir or a protease inhibitor.  After the FDA allowed Epivir and various protease inhibitors on to the market, HAART became generally available.  The graph shows as the use of HAART increased from 0% to over 50% population of California AIDS patients, quarterly survival for these patients fell from 7% to 2%.

While this is compelling evidence that HAART had some causal effect on the survival of AIDS patients it may be harder to determine the extent of that effect or how that effect may vary across patients.  One thing that is clear in the graph is that there is some decrease in quarterly mortality prior to HAART becoming generally available.  That decrease may have been due to the availability of HAART to the clinical trial population which for AIDS peaked at 30% of the population (see previous post).

It is interesting to note that results from one of the first randomized control trials on the effect of combination therapy was published in September 1997.  By the time of publication, HAART use had already hit 50% and quarterly mortality had already fallen to 2%.