Please note: The following article is intended to present facts on the process of post-hoc analysis. It is not intended to express a position regarding the regulatory approval of drugs to be marketed for the treatment of ALS.
In order for a clinical trial to ultimately be considered successful, the treatment in question must demonstrate some measurable benefit for the participants who are taking it. This may seem relatively straightforward, but clearly measuring a clinical benefit can be very challenging, especially in a disease as diverse as amyotrophic lateral sclerosis (ALS).
When designing a clinical trial, the investigators must set out clear goals – called “endpoints” –that set benchmarks for the minimum treatment benefits observed to establish that the treatment is effective. They must also carefully select the demographics of the people who will participate in the trial to make sure that they will be able to conclusively determine that the results they observed were because of the treatment being investigated – and not due to random chance.
A trial in which the participants receiving the test treatment do not meet these pre-defined endpoints is generally considered to have “failed.” However, even a failed trial can produce important data. The act of looking at the data from a trial or other experiment in new ways after the study’s conclusion is called “post-hoc analysis.” These after-the-fact analyses can reveal important information for planning further studies about a drug – with limitations.
How Post-Hoc Analysis Works
“When a trial is done, we have this data set of people that have gotten the drug, people who are on placebos or standard of care,” says Dr. Lori Chibnik, a biostatistician in the Mass General Department of Neurology and assistant professor at the Harvard T.H. Chan School of Public Health. “Even if our primary endpoint was not achieved, there's still information in that data that can help us as we think about the next trial we might want to do. It could be that we want to test this drug in a different population. It could be that we're thinking about other drugs that we might want to be testing. Post-hoc analyses are questions that we try to answer with our data after the study had finished and was not the intent of that particular study”
In a post-hoc analysis of a clinical trial, researchers will often further divide data to see if the drug had benefits for certain groups. Consider an example where the initial data for a trial has shown no benefit in the group that received the drug when compared to a placebo group. A researcher conducting a post-hoc analysis might look at only people who are under a certain age, or only people of a certain sex, and find that there is evidence that in this smaller group, that participants did appear to decline slower or survive longer.
Limitations of Post-Hoc Analysis
Researchers must approach evidence found through post-hoc analysis very carefully. Every time researchers further divide their study population, they are further increasing the likelihood that the effects they are observing are due to chance, according to Dr. Chibnik.
“Say you have 60 in a trial who had a drug and 60 who had a placebo,” she says as an example, “And one person in the placebo group declined very fast and had a very extreme case. That one person isn't going to pull all the placebo group down. However, if you're doing a sub-group analysis and you only have 10 people on drug, 10 on placebo, and one of those just happens to be somebody who has a really extreme case, it's going to make a huge difference.”
“The smaller the subgroups,” Dr. Chibnik continues, “the more likely that all it would take is one person to really influence the data. And then every time we further divide the data, we increase the chance that what we're seeing is just a figment of our data and not and not representative of the population of ALS patients.”
Post-Hoc Analysis as Support for Further Trials
Historically, across most disease indications, it has been atypical that a post-hoc analysis of the data from a trial that failed to meet its endpoints has offered sufficient evidence to bring a treatment to FDA approval. However, exceptions may be emerging in terminal diseases like ALS, where there is such a critical unmet need. That said, where these results are often useful, is in helping investigators focus their approach for future studies, and potentially creating trial designs that are more likely to succeed.
“The more analyses that are done post-hoc, the greater chance of finding results by chance – meaning that they are not true results,” says Dr. Merit Cudkowicz, the Chief of Neurology and Director Sean M. Healey & AMG Center for ALS at Mass General Hospital. “It is always important to repeat results prospectively to best interpret post-hoc analyses. Many if not the majority of post-hoc analyses in trials do not repeat when tested in new trials with these subsets of interest. The utility of post-hoc analyses are to learn more and have new hypotheses. These new hypotheses can warrant then further testing to confirm or better understand.”
Better Measurements for Better Data
At the ALS Therapy Development Institute (ALS TDI), we are working to improve the way trials are run. Today, nearly all ALS trials must rely on clinical observations and subjective questionnaires like the revised ALS Functional Rating Scale (ALSFRS-r) to track disease progression. The ALSFRS-r can be an effective tool, but is limited by the fact that it is a subjective measure of a person’s abilities – it relies on the observer’s assessment about how the disease is affecting a person’s ability to complete the tasks being evaluated by the survey.
Through our ALS Research Collaborative (ARC), ALS TDI researchers are searching for objective measures of disease progression, including biomarkers and artificial-intelligence powered measures of disease progression. With more sensitive measures of disease progression such as these, trial investigators would be able to see the effects of potential treatments quicker and more effectively – and, in effect, reducing the need for further trials that might be informed by post-hoc analyses.
“We are looking prospectively for sensitive indicators of disease presence or absence and of disease progression,” says Dr. Alan Gill, ALS TDI’s Vice President of Research. “More sensitive measures could shorten the length of time we need to treat during a clinical trial and get to the point sooner. We and others will need to prove to the FDA that our progression indicators are, in fact, sensitive and accurate. But when we do, trials will shorten and more drugs will be approved.”
In late January of 2022, ALS TDI and our collaborators at Google made an important step toward making this goal a reality for people living with ALS with the preprint publication of a paper describing their research to create artificial intelligence tools for scoring ALS-related symptom severity. To learn more about their findings, and how they hope to spur the further development of reliable and objective digital measures for ALS progression, click here.
What to do next
