Back to the Top
Background:
Typically the ability of a PK prediction method is evaluated on a
Validation data set using the parameters:
MPE - SUM(Observed-pred)/N/Observed*100%
RMSE - SQRT ( (Observed-Pred)^2 / N))
The mean prediction error (MPE) is easy to understand. However, it
is difficult for me to assign a meaning to a particular value of
RMSE.
In ANOVA or linear regression method, one way to test the consistancy
of conclusions is to apply the model and coefficients to a separate
validation data set. A MSPE (mean square predication error) is then
calculated as (Observed-Pred)^2/N which is the same as RMSE^2.
The MSE in ANOVA is analogous to a variance. For ANOVA validation,
one compares MSPE to MSE. If the two are similar, the MSE from
the ANOVA model provides a good estimate of variance. If MSPE>MSE,
then MSPE is a better estimate of the variance.
Question:
Since MSPE is a good estimate of variance, then RMSE should be a good
estimate of standard deviation. Is it OK to present RMSE as a CV%,
ie. RMSE/Parameter Mean * 100%? This presentation would be much more
meaningful.
Back to the Top
I effectively refer to RMSE while considering that it has the dimension of
a standard deviation. However, unlike the MSE in ANOVA or regression
analysis, it estimates not only the average spread around values provided
by the model, but it also includes a component of lack-of-precision or
misreproducibility between a prediction method and the corresponding
"reality". This has been extremely well explained by Lew Sheiner and Stuart
Beal in one of the most widely cited paper in the PK literature
(J.Pharmacokinet.Biopharm.1981;9:503-12).
As the predicted measures frequently represent concentrations in the field
of PK, and the concentrations frequently follow a skewed distribution,
there would be some advantage in expressing the lack-of-precision in
relative rather than absolute terms. This is equivalent to consider CVs
more informative than SDs. To derive an index of relative lack-of-precision
from the RMSE, I apply the following approach, based on the
"transform-both-sides" idea :
1. Transform both the predicted concentrations and the observed concentrations
into Logs,
2. Calculate the MPE and RMSE.
(For example, consider finding values of 0.20 for MPE and 0.50 for RMSE)
3. Transform it back by taking the exponential
(in this example you will find 1.22 and 1.65 respectively)
4. Substract 1 from the result and express it as a percentage
(like one would say "the Dow Jones has increased by 5%" rather than
"the Dow Jones has been multiplied by a factor of 1.05").
You obtain a mean prediction error expressed in percent (in the example,
you get an average overprediction of 22%), and a relative prediction
imprecision expressed in percent (in the example 65%). If your
concentration data are distributed more or less according to a log-normal
distribution, the interpretation of the latter term sounds like : "if you
try to predict 100 measurements with your model, then they would lie less
than 65% away from the reality in about 68 cases".
Notice that this precision interval is asymetrical, [ -39% to +65% ]. Or
said differently, instead of taking the arithmetical RMSE to state that
your predictions are true 'plus or minus' a certain number of times the
RMSE, you consider a "Geometrical RMSE" to indicate that your predictions
are true 'times or divided-by' a certain number of times the GRMSE (this
certain number of times is given by the Student's t distribution). This is
in accordance with the skewed distribution assumed for the measurements and
prediction errors.
(in the above example the interval ranges from 1/1.65 - 1 to 1*1.65 - 1).
I think that this approach is more exact than simply taking the traditional
RMSE and dividing it by the average of predicted values, especially when
these values cover a wide range.
I hope this helps
Thierry BUCLIN, MD
Division of Clinical Pharmacology
University Hospital CHUV - Beaumont 633
CH 1011 Lausanne - SWITZERLAND
Tel: +41 21 314 42 61 - Fax: +41 21 314 42 66
PharmPK Discussion List Archive Index page
Copyright 1995-2010 David W. A. Bourne (david@boomer.org)