Statistical significance does not necessarily reflect clinically meaningful changes.

Some time ago I published the post you see below on LinkedIn. It was about a quantitative review where researchers evaluated the effects of interactive video games on functional balance and mobility in post-stroke individuals. In summary, Ferreira and colleagues came to the conclusion that interactive video games may lead to significant improvements on functional balance in individuals post-stroke. But are these findings really from clinical relevance?


Fig 1. Screenshot of the blog post.


What we can see in the picture above is that the overall mean difference between intervention and control group is about 2.24 Points (CI 0.45 to 4.04, p = 0.01) on the Berg Balance Scale (BBS). And what we also can see is that under assumption of the validity of the underlying meta-analytical methods (That are not reported!) these outcome differences are statistically significant . However, please note that statistical significance does not necessarily reflect a clinical significance! To find out whether the reported results actually describe a clinically relevant difference between the groups, we don't need to focus on statistics, we need to focus on the so called Minimal Clinical Important Change (MCIC)!


The MCIC is defined as the minimal change in the score that is meaningful for patients. Different approaches can be used to determine the MCIC for a clinical assessment. One is to estimate the mean change in score in patients who actually report to have improved. Another approach is to use Receiver Operating Characteristics curves (ROC) to define a cutoff point that best discriminates between patients reporting or denying any improvement. 


The purpose of this study was to determine the MCIC and its relation to the minimal detectable change (MDC) for BBS in patients early after stroke. Here the researchers used a receiver operating characteristic (ROC) curve to calculate the cut-off value for the MCIC (see picture below).


Fig 2. ROC courve generated with chance in Berg Balance Scale.


Fifty-two patients (mean age of 78.7, SD 8.5 years) were included in this investigation. The ROC analysis identified a MCIC of ≥6 BBS points, while the MDC was 5.97 BBS points at the 80% confidence level.


In conclusion we can say that a minimum change of 6 points on the BBS can be considered an important change for patients in the sub-acute phase after stroke! This means for the interpretation of the result that interactive video gaming may lead to statistically significant improvements on functional balance in stroke patients, but from a clinical point of view this is of little relevance as patients do not perceive these improvements (2.24 BBS Points!) as such.


What do we learn from this example? Statistical significance never means that a result is also important from a clinical perspective! Keep that always in mind and check study results always in both directions!