Bayesian approach for design and analysis of medical device trials in the era of modern clinical studies

Borrowing information

One of the most prominent characteristics for the Bayesian approach is its ability to incorporate prior information through the specification of prior distribution. If prior information is appropriately used, the Bayesian approach can substantially reduce the sample size and study duration. Medical devices develop more rapidly than drugs. The prior information comes naturally from the study of previous versions of the device with similar action mechanisms or the study accomplished in other regions. Borrowing information have been particularly successful to assess effectiveness and safety of cardiovascular devices [12], owing to their fast update speed. There have been 3 cardiovascular devices approved by FDA through the Bayesian information borrowing methods (Table 1). For example, to assess the primary effectiveness endpoint (i.e., the change in peak V02 at 24 weeks from baseline) of an implantable impulse generator [13], a Bayesian model was employed to construct prior distribution obtained from a previous study (FIX-HF-5) 229 patient subgroup. The posterior probabilities were >0.91, meaning that there is a probability of 91 % that the device group was superior to the control group. In late section, we will review some commonly used Bayesian information-borrowing methods.

The likelihood principle and predictive distribution

The likelihood principle is not only the foundation of the frequentist approach but also the core of the Bayesian approach. This principle says that all information about the endpoint obtained from a clinical study is included in the likelihood function [14, 15]. Therefore, the posterior distribution, obtained by combining the prior distribution with the likelihood, should be the basis for all Bayesian inferences about the parameters.

The posterior distribution renders the Bayesian approach great flexibility in study design. Under the Bayesian paradigm, the predictive distribution of the future observations can be generated by first sampling model parameters from their posterior distribution and then, conditional on the parameters, sampling the future observations from the likelihood function. In contrast, for the frequentist approach, the probability of the future observations can only be obtained by fixing the model parameters, which ignores the uncertainty associated with the parameter estimation. Posterior and predictive distributions have been widely used in Bayesian clinical studies, such as making go/no-go decisions, and using covariates to predict clinical outcomes [16, 17]. For example, the Bayesian optimal phase 2 (BOP2) design [18], which makes go/no-go decision based on posterior probabilities, have been used in over 100 on-going clinical trials [19]. There have been 16 medical devices approved by FDA through the Bayesian predictive distribution (Table 1). For example, a Bayesian predictive modeling was used to estimate the 3-year mortality rate and compared to a performance goal (PG) of 12.9 % in a Drug-eluting peripheral transluminal angioplasty catheter clinical study [20].

Exchangeability

The Bayesian approach often assumes that participants within a study are exchangeable, and the studies are also exchangeable with each other [21]. If the probability of observing any set of endpoints within these participants, subgroups, or studies is invariant to any reordering, these participants, subgroups, or studies are considered exchangeable. In this case, the data from these studies could be directly combined. However, this is often not the case in practice. Observations from different studies are often not exchangeable due to different regions, different centers, or different researchers, even though the protocols seem similar with each other. Moreover, when the new version of a device is expected to be superior to previous versions, the current study is typically not exchangeable with previous studies. Therefore, it is necessary to consider conducting participants regularization, choosing appropriate borrowing information methods, and quantifying the amount of borrowing. In addition, the current study and previous studies may not be exchangeable due to differences in baseline covariates, although they might be exchangeable after conditioning on the covariates [22].

Bayesian adaptive design

The FDA guidance discussed two major approaches: one is borrowing information from previous studies, which is introduced in the section of “Borrowing information”; another is to use Bayesian adaptive design [9]. Bayesian adaptive study is usually designed with no prior information but rather relying on accumulating data collected within the current study to potentially make preplanned changes during the course of the study. For example, during a Bayesian adaptive study, the sample size could be modified according to accumulating data while maintaining the targeted power, which may result in shorter study duration and smaller sample size [23]. This sample size re-estimation can be based on the predictive distribution, accounting for not only the subjects already enrolled in the trial, but also the subject yet to be recruited. Moreover, Bayesian adaptive design could construct likelihood predictive models to predict endpoints at later time points (unobserved outcomes) based on the information obtained at earlier follow-up time points (observed outcomes). The primary outcomes using time-dependent intermediate ones are usually modeled by piecewise exponentials [24].

Bayesian adaptive design is particularly suitable for cardiovascular and orthopedic device assessment [25], [26], [27], [28]. These kinds of medical device usually have a slow accrual rate relative to the patient follow-up so that is very expensive to conduct. Therefore, there are adequate time to predict and make adaptations based on the several interim follow-up measurements to make effort to shorten the study duration and reduce the sample size. There have been 9 medical devices approved by FDA through the Bayesian adaptive design (Table 1). The safety and effectiveness assessment of an electrosurgical ablation system was a typical Bayesian adaptive design, which fully used the Bayesian approach to evaluate primary outcomes, including interim analysis, sample size re-estimation, and final decision [27]. The minimum total sample size was 50 and the maximum was 100. At each interim analysis, the predictive probabilities of meeting the primary safety and effectiveness endpoint were calculated using information from the subjects with known outcomes, in combination with a beta-binomial distribution for modeling the transition from either baseline or 3-month outcomes to the 6-month outcomes. Whether to stop patient accrual, stop the study for futility, or continue enrolling subjects was decided by the predictive probabilities.

Bayesian diagnostic design

The Bayesian approach could also be applied in the assessment of medical diagnostic devices [29]. Under the framework of Bayesian diagnostic design, the disease prevalence is considered as prior probability (pre-test probability), and the predictive value of the test is considered as posterior probability (post-test probability) [30]. The probability of the positive test results in truly positive subjects is translated to sensitivity. Similarly, the probability of the negative test results in truly negative subjects is translated to specificity. Furthermore, the positive predictive values (PPV) and negative predictive values (NPV) of the device could be defined by prevalence, sensitivity, and specificity. Indeed, Bayesian diagnostic design is still based on the theory of using prior distribution to predict posterior distribution.

Bayesian multiregional design

The International Conference on Harmonization (ICH) Guideline E5 (1998) proposed multiregional clinical trial (MRCT) design for accelerating the development of innovative drugs and medical devices [31]. MRCT design has advantages of faster enrollment rate, less cost, and providing an opportunity of simultaneous multiregional registration. However, the overall effect may not be consistent from region to region due to the variability among regions. It is difficult to conduct statistical evaluation of the safety and effectiveness for the intended population, especially the region by treatment interaction without statistical significance [32, 33]. Compared to drugs, medical devices tend to be with smaller sample size. MRCT of medical devices are often planned with sufficient subjects to indicate an overall effect in the primary endpoint, thus there may not be adequate power to indicate a statistically significant result at the usual significance level within a specific region. The above problems in effect statistical evaluation could not be solved well by the frequentist approach.

Bayesian hierarchical model could borrow information across different layers, which is appropriate to assess the overall and regional treatment effects and determine the optimal sample size in multiregional medical device study. Chen evaluated regional treatment effects in a multiregional trial through a Bayesian approach [34]. The overall treatment effect was used as prior and conditioned on the observed regional treatment effect to construct the posterior distribution of the regional treatment effect. Furthermore, Bayman discussed the statistical operation and sample size for multiregional clinical studies under a Bayesian hierarchical model [35]. Compared to the frequentist approach (i.e., conventional subgroup analysis), the use of Bayesian hierarchical model in multiregional studies is more powerful because of its ability of borrowing information across different regions, especially existing regional variability. However, the larger the variation the less is the borrowing of information across regions [36]. In addition, the possible underlying caused should be analyzed carefully, when the significant differences exist across regions. The particular regions with large heterogeneity might be excluded from the total analysis.

Bayesian label expansion study

There is a growing interest in leveraging real-world data (RWD) in medical product development. Real-world evidence (RWE) has been used for registrations since the FDA issued the guidance about using real-world evidence in regulatory decisions for medical devices [37]. One of the applications of RWE is to conduct label expansion studies. The Bayesian approach was usually used in these studies benefiting from its ability of borrowing safety information from registered indication to the new indication. That is to say, safety data from real-world use of the device for the approved indication is used to build a prior distribution for the safety assessment of the new indication. There have been several successful experiences in medical device label expansion studies by the Bayesian approach [20, 38]. For example, a drug-eluting coronary stent to the new indications of diabetes patients borrowing information from four real-world databases through a Bayesian hierarchical model [20]. With the rapid speed of accumulating RWD, Bayesian label expansion studies will be more and more widely used in the future registration.

Bayesian FDA submission

Medical devices approved by FDA involving the Bayesian approach from 1998 to 2022 is shown in Table 1. The list is based on the Summaries of Safety and Effectiveness (SSEDs) of the Premarket Approvals (PMA) Database. In general, there have been 47 medical devices approved by FDA which relied on the Bayesian design or Bayesian statistics. Wherein 15 medical devices were approved during 1998–2010. An additional 32 medical devices were approved since FDA published the guidance about the use of Bayesian statistics in medical device clinical trials in 2010. There were 19 (40.4 %) medical devices belonging to cardiovascular and 15 (31.9 %) medical devices belonging to orthopedic because the Bayesian approach could make these medical devices with slow accrual rates and long follow-up much more flexible. Bayesian adaptive study (9, 19.1 %) was the most commonly used designs in medical device clinical trials.

Concurrent borrowing methods

Concurrent borrowing is applied in the master protocol. It is usually used in the study design with multiple parallel arms, such as basket trial design and platform trial design, to borrowing information within multiple sub studies. These arms are of equal importance and are analyzed simultaneously without a chronologic order. Bayesian hierarchical model (BHM) and its extensions and multisource exchangeability models (MEMs) are commonly used in concurrent borrowing. The I-SPY2 trial was designed as a platform trial that used a BHM to adaptively borrow information between running arms [79].

Nonconcurrent borrowing methods

In the era of modern clinical studies, it has become a new trend to use large-scale real-world clinical data and high-quality completed medical data as external data into clinical studies. Nonconcurrent borrowing incorporates historical information from external data into the design of the current study, which offers the possibility of substantially reducing sample size. The paradigm of nonconcurrent borrowing only has one primary study, and others are considered as supplementary studies to provide historical information. Historical information is extracted to construct informative prior, which is further combined with the likelihood function of the current data to make inference and study decisions. Power prior (PP), commensurate prior (CP), elastic prior, meta-analytic-predictive prior (MAP), self-adaptive mixture (SAM) prior, and multisource exchangeability models (MEMs) are examples for nonconcurrent borrowing. In addition, the innovative propensity score (PS)-integrated priors have been proposed, which are considered as an efficient method to deal with the heterogeneity within patient-level covariates.

Application scenarios of information borrowing methods

Application scenarios of information borrowing methods have been the hot topic discussed by researchers all over the world. The recommendation for application scenarios on borrowing information methods are shown in Figure 1.

Figure 1:

Application scenarios of the Bayesian information borrowing methods. BHM, Bayesian hierarchical model; CBHM, calibrated Bayesian hierarchical model; CLBHM, clustered Bayesian hierarchical model; BaCIS, Bayesian hierarchical classification and information sharing; BCHM, Bayesian cluster hierarchical model; MEMs, multisource exchangeability models; PP, power prior; MPP, modified power prior; CPP, calibrated power prior; CP, commensurate power prior; MAP, meta-analytic-predictive prior; RMAP, robust meta-analytic-predictive prior.

Concurrent borrowing methods are useful for applications involving multiple parallel arms, such as basket and platform trials. To choose an appropriate concurrent borrowing method, we should consider exchangeability, statistical performance, types of outcomes, and the number of effective arms. When all arms are homogeneous, the BHM approach is recommended because it shows greater power than other methods. In practice, it is more common that arms are heterogenous or could not be determine the exchangeability. CBHM with a function of heterogeneity measurement, and CLBHM, BaCIS and BCHM with dynamic clustering are recommended. CBHM shows better performance in controlling type I error, whereas CLBHM, BaCIS and BCHM often yield higher power. In addition, when the outcome is ordinal, BCHM tends to gain greater power. MEMs are recommended when the majority of arms are expected to be effective. In terms of balancing type I error and power gain, CLBHM often outperforms BaCIS, BCHM and MEMs, and also is much easier to implement.

Nonconcurrent borrowing methods are appropriate for applications intended to borrow information from external data or supplementary trials. Again, to choose an appropriate method, we should consider exchangeability, statistical performance, and the number of historical trials. When heterogeneity exists, PP and MAP lead to substantial type I error inflation. In this case, their extensions should be considered to obtain better type I error control. For example, CPP provides a stricter criterion for controlling type I error which is recommended to apply in bioequivalence trial. CP and RMAP place more weight on power gains. SAM prior has the best overall performance in improving power and controlling type I error, due to its ability to adaptively adjust the mixture weight based on the prior-data conflict. In addition, when the number of historical trials is small, it is not appropriated to apply MAP and RMAP, and MEMs should be considered.

Moreover, when patient-level covariates are different among different data sources, the PS-integrated priors approach are recommended. PS-MAP and PS-RMAP are recommended to deal with multiple external data sources because of the ability of type I error controlling [16, 17]. In particular, PS-RMAP considers heterogeneity not only between the current study and the external data sources but also between the external data sources themselves. However, the simulation study of Lu et al. showed that their extended PS-PP approach could have better performance than others in type I error controlling when multiple external data sources exist [102]. Therefore, the more comprehensive simulation studies are needed to assess the application scenarios of these available PS-integrated approaches.

Finally, to choose and use any borrowing information method, we should conduct comprehensive simulation study to evaluate its operating characteristics, e.g., the type I and type II error rates [8]. It is critical to balance power gain and type I error control. Justifying the choice of the borrowing information also requires a great deal of communication between the medical device sponsors and the regulatory agency.

Bayesian statistical software

Bayesian statistical software includes R, Stan, SAS, WinBUGS, Python and et al. R is more available and updates more timely than other software. Moreover, R has the interface to Stan and WinBUGS that provides more comprehensive and friendly operation to researchers. Therefore, R has become the most commonly used software in practice. Table 3 summarizes the available R packages and their functions. Stan is for Bayesian modeling and inference that primarily uses the No-U-Turn sampler (NUTS) to obtain posterior simulations. The R package RStan provides the function of Stan allowing one to conveniently fit Stan models from R and access the output, including posterior inferences and intermediate quantities such as evaluations of the log posterior density and its gradients [107]. The R package SAMprior can be used to dynamically borrow information from single to multiple external RWDs based on the SAM prior method [96]. The R package psrwe [112] and RBesT [113] focus on the PS-integrated methods for incorporating external RWD in clinical studies and synthesize the evidence.

Current situations

The FDA and Bayesian statisticians have been working collaboratively to develop Bayesian methodologies for medical device clinical studies since 1998. Nowadays, the Bayesian approach for design and analysis of medical device clinical studies has increased dramatically. This paper summarizes Bayesian designs which are commonly used in the medical device regulatory setting, and reviews several Bayesian information borrowing methods in the era of modern clinical studies and discusses their appropriate application scenarios.

The Bayesian approach provides greater flexibility in study designs compared to the frequentist approach. Bayesian adaptive study design could adjust design elements (e.g., go/no-go, sample size, and population) based on the accumulating data. There have been 9 medical devices with the Bayesian adaptive design acquiring the FDA approval according to the PMA Database since 1998, mainly in cardiovascular and orthopedic regions [25], [26], [27, 57], [58], [59, 61, 68, 74]. They tended to have slow accrual rate but need long follow-up, which is particularly suitable to use the Bayesian adaptive design to reduce sample size and shorten the length of studies. Moreover, the Bayesian approach also has great potential in multiregional studies. The regional variability could be captured by the Bayesian hierarchical model, while allowing information borrowing across regions. This might be difficult to achieve by the frequentist approach. In addition, with the rapid accumulation of RWD in the era of modern clinical studies, the Bayesian approach has been used for label expansion benefiting from its borrowing safety information from registered indication [20, 38]. The above applications in the Bayesian approach have unprecedentedly facilitated medical device registrations.

With the establishment of large cohorts and databases in the era of modern clinical studies, abundant data sources provide opportunities to borrow information to improve trial efficiency. In this paper, we discuss several commonly used information borrowing methods in both concurrent and nonconcurrent borrowing and their appropriate application scenarios. The specification of the borrowing strength parameter is challenging for many information borrowing methods. Numerous researchers are developing innovative methods to achieve more dynamic or adaptive information borrowing to offer better bias and type I error control. From the point of view of regulator, empirically specifying and data driven are the two common ways to determine the borrowing strength parameter. In the empirically specifying approach, communication with clinical investigators and regulator or reference to published studies could obtain the recommended values. In the data driven approach, estimation according to the heterogeneity [85] or model averaging/selection to fit the value [114] may be considered. As for the choice of information borrowing methods, though we discuss their recommended application scenarios, sufficient simulations are also needed to examine the statistical performance to provide the methodology basis for the discussion with regulator.

There are several limitations in the Bayesian approach. The pure Bayesian statistics do not include frequentist operating characteristics, such as type I error and power. Therefore, it is challenging to justify a Bayesian study in the regulatory setting. In practice, the Bayesian approach is used as an expansion tool for the frequentist approach in most cases. Hybrid Bayesian is based on the principles of Bayesian statistics, and combined with the evaluation of frequentist statistical operating characteristics. It could realize flexible design and analysis, and ensure more patients receiving effective treatments from an ethical perspective without sacrificing the integrity of clinical studies and the necessary statistical operating characteristics.

Future and challenges

With the rapid development of medical devices, there has been an increase in the use of Bayesian designs and analysis for regulatory purposes.

The Bayesian approach integrated with propensity score (PS) will play an important role in real-world evidence (RWE) synthesis. It is very common that patient covariates are different in different studies or databases. PS-integrated prior could not only exert the advantages of information borrowing from Bayesian models but also minimize bias from multisource external data. Researchers are developing various methods that integrated PS appropriately into informative prior based on the available methods [14]. With the development of real-world study (RWD) in the era of modern clinical studies, the role of the Bayesian approach integrated with PS on the bias controlling will receive more and more attention.

The Bayesian approach has the potential use in the mock trial design under the virtual patient framework. In 2015, the Medical Device Innovation Consortium (MDIC) virtual patient working group cooperated with FDA stating the method of assessing fatigue fracture in a hypothetical new ICD lead by a mock trial design. Virtual patients were generated from in-silico models of lead failures [115], and prior information was provided by phantoms [116].

Another potential use of the Bayesian approach is incorporating with artificial intelligence (AI)/machine learning (ML) in diagnostic device clinical studies. Compared to almost non-Bayesian ML models, Bayesian ML models could automatically provide uncertainty quantification of model output through the posterior distribution [117]. However, the current computational algorithms for fitting fully Bayesian ML models have not been satisfactory [118]. This will be the next research direction of the relative region.

The National Institutes of Health (NIH) issued the Adaptive Designs Accelerating Promising Trials into Treatments (ADAPT-IT) funding to encourage the Bayesian medical device studies [119]. The National Science Foundation also considered nonparametric Bayesian methods as a great development and Bayesian computation with complex modeling as useful in a wide range of applications [120]. Moreover, the Bayesian approach in evidence-based medicine (EBM) are mentioned and emphasized in medical student teaching [121].

The future for the Bayesian approach in total medicine and medical devices is bright. Using the Bayesian approach has great potential to accelerate the development of innovative medical devices and their accessibility to patients for disease diagnoses and treatments.