- On 4 Nov 1997 at 09:57:31, "David Nix" (nix.at.Pharmacy.Arizona.EDU) sent the message

Back to the Top

Background:

Typically the ability of a PK prediction method is evaluated on a

Validation data set using the parameters:

MPE - SUM(Observed-pred)/N/Observed*100%

RMSE - SQRT ( (Observed-Pred)^2 / N))

The mean prediction error (MPE) is easy to understand. However, it

is difficult for me to assign a meaning to a particular value of

RMSE.

In ANOVA or linear regression method, one way to test the consistancy

of conclusions is to apply the model and coefficients to a separate

validation data set. A MSPE (mean square predication error) is then

calculated as (Observed-Pred)^2/N which is the same as RMSE^2.

The MSE in ANOVA is analogous to a variance. For ANOVA validation,

one compares MSPE to MSE. If the two are similar, the MSE from

the ANOVA model provides a good estimate of variance. If MSPE>MSE,

then MSPE is a better estimate of the variance.

Question:

Since MSPE is a good estimate of variance, then RMSE should be a good

estimate of standard deviation. Is it OK to present RMSE as a CV%,

ie. RMSE/Parameter Mean * 100%? This presentation would be much more

meaningful. - On 5 Nov 1997 at 11:35:50, "Thierry.Buclin" (Thierry.Buclin.at.chuv.hospvd.ch) sent the message

Back to the Top

I effectively refer to RMSE while considering that it has the dimension of

a standard deviation. However, unlike the MSE in ANOVA or regression

analysis, it estimates not only the average spread around values provided

by the model, but it also includes a component of lack-of-precision or

misreproducibility between a prediction method and the corresponding

"reality". This has been extremely well explained by Lew Sheiner and Stuart

Beal in one of the most widely cited paper in the PK literature

(J.Pharmacokinet.Biopharm.1981;9:503-12).

As the predicted measures frequently represent concentrations in the field

of PK, and the concentrations frequently follow a skewed distribution,

there would be some advantage in expressing the lack-of-precision in

relative rather than absolute terms. This is equivalent to consider CVs

more informative than SDs. To derive an index of relative lack-of-precision

from the RMSE, I apply the following approach, based on the

"transform-both-sides" idea :

1. Transform both the predicted concentrations and the observed concentrations

into Logs,

2. Calculate the MPE and RMSE.

(For example, consider finding values of 0.20 for MPE and 0.50 for RMSE)

3. Transform it back by taking the exponential

(in this example you will find 1.22 and 1.65 respectively)

4. Substract 1 from the result and express it as a percentage

(like one would say "the Dow Jones has increased by 5%" rather than

"the Dow Jones has been multiplied by a factor of 1.05").

You obtain a mean prediction error expressed in percent (in the example,

you get an average overprediction of 22%), and a relative prediction

imprecision expressed in percent (in the example 65%). If your

concentration data are distributed more or less according to a log-normal

distribution, the interpretation of the latter term sounds like : "if you

try to predict 100 measurements with your model, then they would lie less

than 65% away from the reality in about 68 cases".

Notice that this precision interval is asymetrical, [ -39% to +65% ]. Or

said differently, instead of taking the arithmetical RMSE to state that

your predictions are true 'plus or minus' a certain number of times the

RMSE, you consider a "Geometrical RMSE" to indicate that your predictions

are true 'times or divided-by' a certain number of times the GRMSE (this

certain number of times is given by the Student's t distribution). This is

in accordance with the skewed distribution assumed for the measurements and

prediction errors.

(in the above example the interval ranges from 1/1.65 - 1 to 1*1.65 - 1).

I think that this approach is more exact than simply taking the traditional

RMSE and dividing it by the average of predicted values, especially when

these values cover a wide range.

I hope this helps

Thierry BUCLIN, MD

Division of Clinical Pharmacology

University Hospital CHUV - Beaumont 633

CH 1011 Lausanne - SWITZERLAND

Tel: +41 21 314 42 61 - Fax: +41 21 314 42 66

Want to post a follow-up message on this topic? If this link does not work with your browser send a follow-up message to PharmPK@boomer.org with "RMSE" as the subject

PharmPK Discussion List Archive Index page

Copyright 1995-2010 David W. A. Bourne (david@boomer.org)