Q&A Report: Evidence Synthesis for Sparse Evidence Base, Heterogeneous Studies, and Disconnected Networks

Experts answer top questions from their recent webinar, “Evidence Synthesis for Sparse Evidence Base, Heterogeneous Studies, and Disconnected Networks”.

Can you please expand on credible intervals and how they are interpreted?

A Bayesian approach is probabilistic and therefore the interpretation of 95% credible intervals is much more intuitive than the interpretation of 95% confidence intervals. In contrast to classical statistics, Bayesian approaches are not based on repeated experimentation, and therefore, a 95% credible interval can be interpreted as: “with a 95% probability, a certain value lies within the credible interval”. This statement could not be used in a frequentist (classical statistics) framework. Frequentist frameworks are based on repeated experimentation. Therefore, the interpretation of a 95% confidence interval is not as intuitive. A 95% confidence interval cannot be interpreted in terms of probabilities, but as “if I repeated my experiment 100 times, in 95 times the true population parameter is covered by the 95% confidence interval”.

Has ML-NMR (multilevel network meta-regression) been used in any HTA (health technology assessment) submission?

The National Institute for Health and Care Excellence (NICE) methods review in 2020 says that ML-NMR is the preferred population adjustment approach for anchored comparisons of more than 2 studies and multiple treatment comparisons. We are not aware of an HTA submission where the method has been used yet.

How to differentiate between effect modifiers and prognostic variables?

An effect modifier is a covariate that alters the effect of treatment on outcomes, so that the treatment is more or less effective in different subgroups formed by levels of the effect modifier. Effect modifiers are not necessarily also prognostic variables. A prognostic variable is a covariate that affects (or is prognostic of) the outcome. Effect modifiers are not necessarily also prognostic variables.

How to identify effect modifiers and prognostic variables?

Evidence that a variable is an effect modifier for the outcome in question should be based on empirical evidence, expert opinion, and/or systematic literature reviews. Quantitative empirical evidence could be obtained from regression models fitted to IPD trial data to investigate whether a covariate is associated with outcome, or whether it alters the treatment effect (interaction with treatment variable).

Is there any threshold for effective sample size and if so, how low can the effective sample size go? What is the interpretation of the low effective sample size?

There is no threshold for effective sample size, but low effective sample sizes are an indication that the weights are highly variable due to a lack of population overlap, and that the estimate may be unstable.

Should we match treatment arm with treatment arm and placebo arm with placebo arm in AD trial, or we should do the trial level matching suggested in NICE TSD?

In general, trial level matching can be used unless there is substantial imbalance in key characteristics despite randomization.

You have presented methods using informative priors for between-study heterogeneity parameters. I was wondering whether it is also feasible to assign informative priors directly on the treatment effect. If so‚ how does this work?

Indeed, this is also feasible and requires prior elicitation, either through expert clinician opinion (where experts can be formally interviewed), or through literature which is not used directly in the NMA, such as observational studies, pilot studies, etc. All relevant information to inform the priors would be collected and then pooled to inform the parameters of a suitable distribution. For example, if the prior is assigned an informative Normal distribution, the corresponding mean would be estimated by pooling the evidence collected from expert clinician opinion and additional literature sources. The corresponding precision would be estimated through a simulation approach, ensuring that the posterior resulted in a range of the treatment effect which corresponded to the estimates collected through prior elicitation.