Current philosophical perspectives on drug approval in the real world

2.2.1 Approval process

In the European Union (EU), the European Medicines Agency (EMA) advises the European Commission on drug regulation based on assessments of evidence. At the time at which an assessment regarding immediate market entry is made, EMA focuses on (typically two phase III) RCTs^[4] which supposedly demonstrate beneficial effects. The possibility of adverse reactions caused by the drug is assessed based on these RCTs as well as on other evidence (such as phase 0 to II trials, animal studies, and computer modelling). We here focus on the standard decision procedures, i.e., we are not considering (i) fast track drug approval for serious conditions nor (ii) orphan drugs intended for so few patients that RCTs with sufficiently many patients to achieve an adequate level of statistical power become infeasible.

The Food and Drug Administration (FDA) in the USA used to follow similar decision procedures [16]. The 21st Century Cures Act in the USA, signed into law in 2016, now also allows applicants of market authorisation to present, and the FDA to rely much more on real world evidence (RWE). RWE can refer “to information on health care that is derived from multiple sources outside typical clinical research settings, including electronic health records (EHRs), claims and billing data, product and disease registries, and data gathered through personal devices and health applications” [17].

In essence, the European approach, according to EBM epis ’s heavy emphasis of randomised studies is to subject drugs that hold promise based on previously accumulated evidence to be effective and reasonably safe to two RCTs. In case the RCTs confirm this initial effectiveness assessment (and the drugs are assessed as safe enough; see Section 2.3.1), EMA recommends approval,^[5] which is then granted by the European Commission.

The approach was developed to put an end to exaggerated claims of benefit [14]. It was thus designed with effectiveness assessments in mind. Beneficial effects, if they do obtain, obtain within a well-understood time frame and well-defined parts of the human body. RCTs are an appropriate means to learn about effectiveness: RCTs are expensive^[6] and thus, study not too long time frames; RCTs analyse the specific parts of (patho-)physiology and lend themselves to statistical (meta-)analysis to detect significant effects.

However, practical problems abound even when the issues concerning randomisation mentioned in the introduction can be avoided. First, defining a statistical measure prior to conducting RCTs is not a trivial matter (progressing from determining a target of estimation [estimand], via fixing methods of estimation [estimators] to calculating numerical results [estimate] including sensitivity analyses) [18]. Second, some planned measurements of variables may not occur (e.g. a patient switching to a different treatment or dying) or may be difficult to interpret (e.g. a patient using an additional medication made available for ethical reasons) [18]. Third, there is the hard methodological problem of passing from an efficacy estimation to effectiveness assessments [19–22].

Until recently, reliance on two RCTs was also the essence of the approach in the USA. Not only did the EU and the USA implement similar decision procedures, they also often made the same decision [23]. Given the provisions in the 21st Century Cures Act, these similarities are set to become less pronounced.^[7] To date, there is limited scholarly research on the effect of these provisions on drug regulation [24]. We hence first focus on EMA’s current approach and then briefly consider changes brought about by the 21st Century Cures Act. EMA’s current approach will be referred to as the status quo.

2.2.2 The evidence

How accurately does EMA’s approach assess the effectiveness of drugs? We now present evidence that drug regulation agencies regularly approve drugs with disappointingly low effectiveness.

“By law, the German health technology assessment agency IQWiG (Institute for Quality and Efficiency in Health Care) must investigate the added benefit of new drugs compared with standard care” [25] (the methodology used for these assessment is available on IQWiG’s website [26]). IQWiG’s findings are stark:

Only 54 of the 216 assessed drugs (25%) were judged to have a considerable or major added benefit. In 35 (16%), the added benefit was either minor or could not be quantified. For 125 drugs (58%), the available evidence did not prove an added benefit over standard care for mortality, morbidity, or health related quality of life in the approved patient population. [25]

The same review found that for new drugs for over 100 indications approved by the FDA

superior efficacy on clinical outcomes was confirmed in less than 10% of cases. A higher but still insufficient rate (20%) was shown in a similar publication on cancer drugs. [25]

Wieseler et al’s findings are widely supported. The proportion of new approved drugs that actually improved on existing pharmaceuticals in the real world ranges between 10 and 20% in a number of reviews that each investigated hundreds of cases [27–30].

These findings are widely acknowledged, yet seem to make little impact on methodology. For example, a paper on pain medication with the title: “The Development of New Analgesics Over the Past 50 Years: A Lack of Real Breakthrough Drugs” concludes starkly:

Morphine and aspirin, introduced for the treatment of pain more than a century ago, continue to dominate biomedical publications despite their limited effectiveness in many areas (e.g. neuropathic pain) and multiple serious adverse effects. [31]

Furthermore, a recent editorial in the BMJ is aptly subtitled: “We must raise the bar to ensure real benefits for patients” [32]. We agree – many drugs are being approved with low effectiveness.

Given the aforementioned body of non-systematically obtained evidence mainly comparing the effectiveness of new drugs to the effectiveness of older drugs, we believe that the status quo approves non-effective drugs because those drugs are assessed as efficacious: the status quo regularly approves drugs with disappointingly low effectiveness, we here mean absolute and not comparative effectiveness. This is in the interest of drug manufacturers.

We think that changes to the status quo are required to improve the effectiveness of approved drugs (Section 4).

In the USA, the 21st Century Cures Act also allows drug manufactures to submit RWE supporting their applications for drug approval. Such evidence, which can be presented as the applicants see fit, can only lead to further overestimation of effectiveness. Scholarship on this act is still in its infancy [33,34], but there are the beginnings of some relevant empirical evidence:

We are concerned that the use of RWE for regulatory drug approval will increase uncertainty over the true benefits of therapies we offer to patients. As investigators who work routinely with RWD (Real World Data), we have concerns that two of the highlighted FDA decisions were based on RWE that came from case series of only 14 and 26 patients. Our patients deserve timely access to new cancer therapies; however, they also deserve therapies for which we have strong evidence showing meaningful gains in quality and/or quantity of life. [24]

2.3.1 Approval process

Unlike intended effects, adverse drug reactions (ADRs) can occur after a long time, e.g. years of treatment with olanzapine causes tardive dyskinesia [35], treatment of pregnant women with diethylstilbestrol caused vaginal adenocarcinoma in their pubertal and adult children [36]. ADRs also occur in unpredictable places and may be rare yet deadly (in some cases, 1 fatality in every 10,000 patients [37]). RCTs are typically too short – “often months to 1 or 2 years” [38] – and too small – “often dozens to a few hundred” patients [38]) – to properly inform harm assessments [39–42]). In light of this, EMA, in accordance with EBM epis , bases safety assessments on not only the data from two RCTs submitted for effectiveness assessments, but also emphasises other available evidence. Indeed, the EU is now calling for efforts to amalgamate safety signals such as spontaneous case reports, data-mining, pharmacoepidemiological studies, drug utilisation studies, and non-clinical studies (Directive 2010/84/EU; Regulation (EU) No 1235/2010).

2.3.2 Evidence

Andreoletti and Teira [14 emphasis original] claimed that “market withdrawals provide an empirical benchmark” for assessing approaches. There are several aspects to withdrawal of approved drugs. One simple measure is the number of drugs withdrawn. We agree that determining the withdrawal rate is a sensible and feasible benchmark. It helps us to understand how often bad drugs are falsely granted approval. However, as Andreoletti and Teira note [14]:

Number of drug withdrawals is admittedly a rough index of regulatory success. For instance, there are no universal guidelines, and therefore, there is no perfect international agreement about which drugs should be available [43]. With every caveat in mind, less than 2 percent of new drug approvals by the FDA between 1950 and 2011 were withdrawn [44].

One might think that withdrawal of only 2% of drugs indicates a strong approval process. However, there are reasons to think the reported withdrawal rate is underestimated. For one, drug licensing agencies have recently begun performing paid consulting work for drug manufacturers [45]. As this is a new development, it is likely not to have impacted the numbers of withdrawals of already approved drugs reported in the study of Onakpoya et al. [44].

A stronger reason is that there are cases in which a drug was not withdrawn, but weaker regulatory decisions were taken, e.g. a change of labelling information concerning ADRs. EMA’s current procedures for monitoring effectiveness and safety are described in the study of Brown et al. [46], who “identified and reviewed all EMA post-market approval referrals made for safety and/or efficacy concerns which were evaluated by an assessment committee between 1 January 2013 and 30 June 2017.” The authors found that 83% of referrals led to changes to product information, while 23% led to suspension of withdrawal of market authorisation. These results are echoed in studies on FDA-approved pharmaceuticals: from 2001 to 2010, 32% of novel therapeutics were subject to post-market safety events (withdrawals, boxed warnings, or safety communications) [47]. While withdrawals were rare (3/222), new boxed warnings were not (43/222) – boxed warnings indicate potentially life-threatening or preventable safety events. Last but far from least, there is evidence that a larger share of novel cancer drugs reduce patient safety rather than improve it (45% compared to 15%) [48]. Hence, the status quo is making errors that have severe consequences.

This evidence of continued updating of safety labelling shows that (i) serious pharmacovigilance efforts continue to take place; (ii) unsuspected ADRs continue to emerge, and estimated frequencies of ADRs continue to rise, after approval; and (iii) approved drugs are actually less safe than deemed at the approval stage. One then asks whether these drugs should have been approved in the first place, and whether they should be withdrawn rather than re-labelled.

Furthermore, regulatory actions – in particular withdrawals – occurring at the first sign of trouble after approval make us wonder about the robustness of the approval and withdrawal processes. How can a few case reports alter the evidence base so drastically that regulatory action becomes necessary? This concern is echoed in [49]:

[T]he speed with which…adverse reactions [appear] in the literature following launch…arouses suspicion of lack of transparency in reporting of harms in clinical trials conducted in the pre-approval phases or flaws in the ways in which harms [are] assessed by regulators. This is also supported by the speed with which…analgesics [are] withdrawn from the market following the reports of such reactions. Indeed one such analgesic, pifoxime, was withdrawn in 1976, within 3 months of regulatory approval in France, following reports of serious neuropsychiatric adverse reactions.

It is plausible then that many drugs that should be withdrawn are being kept on the market through changing of labelling information. At the very least, it is clear that many drugs are approved that in actual fact have potentially severe ADRs not picked up at the approval stage.

Recall that it also matters how quickly issues are rectified (A3). Unfortunately, for the status quo, it looks like it does not do well on this count either:

We found 95 drugs for which death was documented as a reason for withdrawal between 1950 and 2013. All were withdrawn in at least one country, but at least 16 remained on the market in some countries. … However, in 47% of cases more than 2 years elapsed between the first report of a death and withdrawal of the drug, and the interval between the first report of a death attributed to a medicinal product and eventual withdrawal of the product has not improved over the last 60 years. [50]

Pharmacovigilance was not a thing 65 years ago [43,51]. This evidence strongly suggests that harmful drugs are often withdrawn much too late. As noted, for all our benefit, some of these drugs should not have been approved in the first place.

Overall, we think that the evidence demonstrates that there is underestimation of ADRs by the status quo beyond the reported 2% drug withdrawal rate. Accordingly, we find that this approach, implemented in the real world, regularly approves drugs that are disappointingly unsafe: there are a large number of ADRs post-approval; the severity of errors is large; the timeliness of withdrawal is inadequate.

We are not aware of articles systematically reporting on the (un-)safty of drugs approved under the 21 Century Cures Act. A previous law change to allow for faster drug approval by the FDA was found to correlate with a decrease in safety of approved drugs [52]. We have no reason to believe that the more recent law change improves safety assessments.

3.1.1 Approach

EBM + is an alternative to EBM because it requires explicit evaluation of evidence of mechanisms alongside and on a par with evidence of correlation [11,98]. The motivation for EBM + is an epistemological thesis made by Russo and Williamson [99]: in order to establish a causal claim in medicine (A causes B), one normally needs to establish that A is appropriately correlated with B, and establish that a mechanism exists linking A and B that can account for the correlation. This thesis is sometimes called the Russo-Williamson thesis (RWT), and sometimes evidential pluralism.

According to EBM + proponents, high-quality RCTs can establish causation because they provide indirect evidence that both a correlation and a mechanism exist, thus satisfying the conditions of RWT [100]. However, the problem EBM + identifies with EBM epis is that often there are enough issues with the implementation of RCTs that only a correlation is established. In such cases, establishing a mechanism can help, as the two lines of evidence reinforce one another: each line of evidence has a characteristic weakness that is addressed by the strengths of the other [101]. Most people outside the EBM + community disagree with the EBM + stance of treating evidence obtained from RCTs as mere correlational evidence. They instead uphold the epistemological importance of randomisation and the better quality evidence it provides.

EBM + also places no restrictions on the kind of method used – causation can be established by evidence obtained from both clinical and mechanistic studies. The methods employed by mechanistic studies are various, but they are the methods of the “basic sciences” (e.g. microbiology, biochemistry, physiology, molecular biology). At best, in EBM epis , evidence from mechanistic studies is taken as always strictly inferior to evidence obtained from clinical studies. But mechanistic studies can provide direct evidence of the existence of a mechanism [100]. According to EBM + , this is enough to establish a mechanism. So, a combination of clinical and mechanistic studies can suffice to establish causation. Recently, the consortium of philosophers and scientists at the heart of the EBM + programme have produced a set of guidelines for assessing evidence from mechanistic studies [102]. This keeps in the tradition common to EBM epis of systematically evaluating evidence, and operationalises the theory behind the approach.

3.1.2 Philosophical analysis

As noted earlier, there are many parties interested in whether a drug is approved. Reliable methods are supposed to be the way to counter this. But some authors argue that the methods of EBM epis are not reliable and are in fact malleable – they can be bent to the will of researchers to produce the results they want – and this goes a long way to explain why the status quo is unsuccessful at drug approval (see, e.g., [103]). [104] argues that EBM + as a tool for causal inference is actually less malleable than EBM epis since (1) mechanistic studies and clinical studies reinforce each other as the strengths of each line of evidence makes up for the deficiencies of the other line of evidence (this is the basic argument for evidential pluralism); (2) mechanistic evidence is (to a large degree) independent from observational or intervention studies with respect to performing the studies, scientific instruments used in the studies, and the methodology used; and, (3) “Mechanistic studies … are often conducted by research teams with different interests to those who carry out association studies (which tend to be carried out by drug companies seeking approval for lucrative new drugs). Thus publication bias, fraud, and industry manipulation are less of a concern for mechanistic studies than for association studies” [104]. For both benefit and harm assessment, we disagree. Instead, application of EBM + to drug approval problems would plausibly lead to at least as bad outcomes as does EBM epis .

Points 1 and 2 above are about the reliability of competing logics of causal inference – Williamson argues that EBM + is more reliable than EBM epis , hence will head off the problems EBM epis faces. Now, one might think that the reliability claim is not true, e.g. it is not clear that independent evidence is more confirmatory. A growing literature on the variety of evidence thesis highlights that less independent evidence is (counter intuitively?) more confirmatory in a variety of cases: see [105–108] for the latest on such cases. However, even if we assume that (1) and (2) make the logic of the EBM + causal evaluation more reliable than EBM epis , it is not clear that (3) is true – the remainder of this analysis is aimed at rejecting the claim that “publication bias, fraud, and industry manipulation are less of a concern for mechanistic studies than for association studies.”

Our argument starts by recognising that in an EBM + world mechanistic studies will be carried out by drug companies seeking approval for lucrative new drugs. It is plausible that non-profit researchers have the monopoly on mechanistic studies only because there is currently no financial incentive to pharmaceutical companies to be in the game. As theorised more abstractly in [70], if the rules (of the drug approval process) change, the behaviour of the stakeholders will likely change. At a basic level, more mechanistic studies means an increased chance of landing on what looks like a mechanism. This line of argument may falter when it comes to the search for harms, which already leans towards an EBM + approach. The latest regulations for assessments of ADRs already explicitly call for the incorporation of mechanistic and other low-level evidence (Section 2.3.1). This implies that (i) strong enough evidence of a positive correlation between drug use and ADR or (ii) a great enough number of case reports can suffice for an assessment that the drug causes an ADR that would lead to drug disapproval. Drug companies would not want to increase the amount of mechanistic studies here. However, what they can do is exploit our often incomplete knowledge of biological mechanisms to argue that causation has not been established [109]. They can do this by producing conflicting evidence where the case is already strong that there is a plausible harm, or they can call into question the existing evidence where the case is weaker.

The proponent of EBM + may argue in turn that strict enough assessment guidelines – such as Parkkinen et al. [102] – may solve this problem. In particular, that just having more evidence does not necessarily mean more mechanisms will be established. There must be evidence that the mechanism operates and this evidence must be good quality. EBM + immediately runs into some problems. One is that companies will often manage to argue that there is enough evidence to establish a mechanism. If establishing a mechanism is a prerequisite for drug approval, then they will put their mind to establishing a mechanism. For one, they have the budget to carry out a great quantity of research, which is also much cheaper than conducting RCTs. Additionally, there are a great number of models for human (patho)-physiology on which to experiment: a great number of human or animal cell lines, different animal species, biomedical imaging technologies, autopsies, and computer simulations. The myriad of strict assessment frameworks available to regulators working in the status quo have not managed to ward off manipulation; why should EBM + be any different? Furthermore, the multiplicity and heterogeneity of methods used to discover mechanisms is a weakness for EBM + here rather than a strength. Approaches to mitigating industry bias may be difficult to implement for mechanistic studies. For example, pre-specifying a trial analysis is straightforward for an RCT, due to its relative simplicity. But it is very difficult to do this for any one mechanistic study, let alone all the different kinds of studies that can be carried out.

A final problem is the blind spot current EBM + guidelines have for avoiding industry bias. Parkkinen et al. include no tools for assessing the extent of industry influence on the production of evidence. This point has recently been made by Howick [110] and is in contrast to the inclusion of such assessment criteria in some status quo frameworks, e.g. GRADE. Clearly, this gap in the guidelines is there because the EBM + group think mechanistic studies are less likely to be subject to industry manipulation. As we have argued, there may be a greater chance of industry manipulation. This problem is exacerbated by the fact EBM + delivers – by design – only qualitative judgements about the relative plausibility of the target causal relationship. One might worry that this leaves EBM + open to the influence of subjective judgements. There is hence no reason to think that mechanistic studies will differ significantly from association studies with respect to bias, fraud and industry manipulation, and we conclude that implementing EBM + would not succeed to any greater extent than the status quo.

Summing up, an implementation of EBM + for drug approval faces plausible problems of applicability. EBM + is, however, not the finished article, and we hope these critical points suggest new avenues for its proponents to explore.

3.2.1 Approach

E-synthesis has been developed for safety assessments [111–117]. It aims to take into account all the available information and not just focus on RCTs, which is crucial for harm assessments in the real world (see Section 2.3.2). Typically, aggregated bodies of evidence are composed of diverse and contradictory safety signals.

In order to carry out this difficult task, design choices are made. α ) E-Synthesis applies Bayesian epistemology which dominates modern analytic philosophy of science. β ) E-Synthesis interprets the Bradford Hill Guidelines [118] as indicators of causation [119]. The basic idea is that an indicator being true increases our belief in the causal hypothesis, while an indicator being false decreases our belief. γ ) E-Synthesis does not take information at face value: every item of evidence is assessed in terms of a number of evidential modulators, which modulate the extent of confirmation. For example, observational studies are assessed according to the number of observations, the duration of the study, the (strength of) sponsorship bias, the quality of adjustment for confounders and stratification, as well as external validity. δ ) E-Synthesis then applies a Bayesian network approach [120] to calculate a posterior probability of a drug causing an ADR given the available and assessed evidence. In order to define the Bayesian network, the causal hypothesis, as well as every indicator of causation, every item of evidence and every modulator of every study^[9] is represented by a variable. Strictly speaking, E-Synthesis is not wedded to any of these design choices, e.g. different indicators, modulators, and/or structures of the Bayesian network, could be used, yet implementing a number of such changes would result in a different approach worthy of a label different than E-Synthesis. We shall use the term E-Synthesis for the originally proposed approach.

E-Synthesis has not been developed for, nor has it ever been applied to, effectiveness assessments.

3.2.2 Philosophical analysis

E-Synthesis is at even earlier stage of development than EBM + : no comprehensive case study has been put forward, and no practical suggestions have been made for incorporating case reports into the evidence aggregation framework, although case reports play a major role in drug safety assessments in the real world [44]. We hence require a greater dose of imagination to envision how this approach would work in the real world.

Compared to the status quo, E-Synthesis holds the promise to systematically seamlessly aggregate evidence of different kinds: RCTs, observational studies, mechanistic evidence as well as case reports. This promise will, however, go unfulfilled, if the requirement to present the total evidence is not met (Section 2.4).

Furthermore, E-Synthesis makes a number of judgements explicit (e.g. assessed values of modulating variables, importance of indicators of causation) and allows their confirmatory roles to be tracked for hypothesis confirmation. This can be a significant improvement relative to the status quo in which safety assessments are – in the end – narrative reviews.

While the capability of explicity can be a major advantage, it also, we think, harbours a grave danger for sensible real world applications of E-Synthesis. Explicity is facilitated by the use of a great number of variables. The specification of the resulting Bayesian network required to calculate a posterior probability of a causal hypothesis of interest is hence carried out by the definition of a great number of probabilities. These probabilities are meant to represent an ideally rational agent’s credences. Unfortunately, no ideally rational agent inhabits our world.

Setting these probabilities involves a good deal of subjective choice. While some think that subjectivity is not much of an issue [121], we are much more pessimistic. Our view is a consequence of the track record of rule exploitation, bending, and breaking, as well as the size of the incentives at stake (Footnote 3). To take some examples of subjective choice, (i) a prior probability of the causal hypothesis of interest must be specified; (ii) the strength of sponsorship bias of every item of evidence must be assigned a probability; (iii) the confirmatory value of all sets of indicators of causation must be formalised by defining conditional probabilities; (iv) all evidence modulating variables must be assessed; and (v) conditional probabilities of all items of evidence given the assessed modulating variables must be determined.

Any way of setting these probabilities in a scientifically defensible way will have to produce them in an objective sense, which would make use of all the other available information. For example, data from previous safety assessments could, in principle, be used to determine frequency information regarding the probabilities of indicators of causation, the strength of sponsorship bias, and the conditional probabilities of items of evidence given assessed sponsorship strengths. Distilling this information in the real world necessitates the solution of a greater number of choice problems, which do not have objectively correct solutions. For example, does one take into account studies concerning: the same/similar adverse event/reaction; same/similar study design; same/similar or even all drug manufacturers; same/similar drug design; same/similar group or researchers? These issues constitute a real world reference class nightmare [122]. Furthermore, there is a great number of ways to operationalise graded notions of the quality of adjustment, blinding, randomisation, and external validity.

We hence believe that every application of E-Synthesis to a drug approval problem would require a great number of choices to be made, where there are no obvious unique correct solutions. This entails that a great dose of subjectivity is required. In turn, this opens the floodgates for drug manufacturers to influence the outcome of safety assessments.

Overall, we believe that implementation of E-Synthesis would not lead to safer drugs but rather to more drugs being granted market approval and unsafer drugs as a consequence. Since the status quo is already under-assessing harms (Section 2.3), an implementation of E-Synthesis in its current form cannot be recommended.

3.3.1 Approach

Before concluding, we now sketch a philosophical approach to drug approval that is neutral on how to draw causal inferences but differs on who ought to make decisions. Libertarian thinkers defend the view that patients experiencing the (good and bad) effects of drugs ought to be the ones who decide which drugs they take [123,124] (see [125] for more historical information on the paternalism vs freedom debate in drug approval).

3.3.2 Philosophical analysis

Teira recently argued that patients suffering ADRs will find it very hard to prove in court that they indeed suffered an ADR and hence are very unlikely to be compensated [126]. This argument is based on significant information asymmetries between plaintiffs and the defending corporation, as well as significant asymmetries to fund legal teams. We agree.

He went on to argue that drug regulation agencies ought to collect the available evidence and present it to the decision makers, the patients. We believe that this is unlikely to happen in the real world, since even the current regime fails to base their assessments on all the available evidence (Section 2.4).

Without access to a trusted impartial entity performing causal inference and decision making, real world patients will often fail to make good decisions, since they (1) typically lack the medical training to properly understand the medical details of studies; (2) are typically not trained in causal inference, and will thus struggle with properly assessing evidential support relations between causal hypotheses on the one side and observational and/or randomised studies on the other side; (3) are often ill and are thus not in a mental state to do difficult judgements, inferences and assessments; and (4) often do not have the time to read all the presented material.^[10]

Overall, we strongly believe that the outcomes of all implementations of an empowered patients approach are significantly worse for patients than the status quo.