# PharmPK Discussion - RMSE

PharmPK Discussion List Archive Index page
• On 4 Nov 1997 at 09:57:31, "David Nix" (nix.at.Pharmacy.Arizona.EDU) sent the message
`Background:Typically the ability of a PK prediction method is evaluated on aValidation data set using the parameters:MPE - SUM(Observed-pred)/N/Observed*100%RMSE - SQRT ( (Observed-Pred)^2 / N))The mean prediction error (MPE) is easy to understand.  However, itis difficult for me to assign a meaning to a particular value ofRMSE.In ANOVA or linear regression method, one way to test the consistancyof conclusions is to apply the model and coefficients to a separatevalidation data set.  A MSPE (mean square predication error) is thencalculated as (Observed-Pred)^2/N which is the same as RMSE^2.The MSE in ANOVA is analogous to a variance.  For ANOVA validation,one compares MSPE to MSE.  If the two are similar, the MSE fromthe ANOVA model provides a good estimate of variance.  If MSPE>MSE,then MSPE is a better estimate of the variance.Question:Since MSPE is a good estimate of variance, then RMSE should be a goodestimate of standard deviation.  Is it OK to present RMSE as a CV%,ie. RMSE/Parameter Mean * 100%?  This presentation would be much moremeaningful.`
Back to the Top

• On 5 Nov 1997 at 11:35:50, "Thierry.Buclin" (Thierry.Buclin.at.chuv.hospvd.ch) sent the message
`I effectively refer to RMSE while considering that it has the dimension ofa standard deviation. However, unlike the MSE in ANOVA or regressionanalysis, it   estimates not only the average spread around values providedby the model, but it also includes a component of lack-of-precision ormisreproducibility between a prediction method and the corresponding"reality". This has been extremely well explained by Lew Sheiner and StuartBeal in one of the most widely cited paper in the PK literature(J.Pharmacokinet.Biopharm.1981;9:503-12).As the predicted measures frequently represent concentrations in the fieldof PK, and the concentrations frequently follow a skewed distribution,there would be some advantage in expressing the lack-of-precision inrelative rather than absolute terms. This is equivalent to consider CVsmore informative than SDs. To derive an index of relative lack-of-precisionfrom the RMSE, I apply the following approach, based on the"transform-both-sides" idea :1. Transform both the predicted concentrations and the observed concentrations   into Logs,2. Calculate the MPE and RMSE.   (For example, consider finding values of 0.20 for MPE and 0.50 for RMSE)3. Transform it back by taking the exponential   (in this example you will find 1.22 and 1.65 respectively)4. Substract 1 from the result and express it as a percentage   (like one would say "the Dow Jones has increased by 5%" rather than   "the Dow Jones has been multiplied by a factor of 1.05").You obtain a mean prediction error expressed in percent (in the example,you get an average overprediction of 22%), and a relative predictionimprecision expressed in percent (in the example 65%). If yourconcentration data are distributed more or less according to a log-normaldistribution, the interpretation of the latter term sounds like : "if youtry to predict 100 measurements with your model, then they would lie lessthan 65% away from the reality in about 68 cases".Notice that this precision interval is asymetrical, [ -39% to +65% ]. Orsaid differently, instead of taking the arithmetical RMSE to state thatyour predictions are true 'plus or minus' a certain number of times theRMSE, you consider a "Geometrical RMSE" to indicate that your predictionsare true 'times or divided-by' a certain number of times the GRMSE (thiscertain number of times is given by the Student's t distribution). This isin accordance with the skewed distribution assumed for the measurements andprediction errors.(in the above example the interval ranges from 1/1.65 - 1 to 1*1.65 - 1).I think that this approach is more exact than simply taking the traditionalRMSE and dividing it by the average of predicted values, especially whenthese values cover a wide range.I hope this helps                                          Thierry BUCLIN, MD                           Division of Clinical Pharmacology                     University Hospital CHUV - Beaumont 633                            CH 1011 Lausanne  -  SWITZERLAND               Tel: +41 21 314 42 61 - Fax: +41 21 314 42 66`
Back to the Top

Want to post a follow-up message on this topic? If this link does not work with your browser send a follow-up message to PharmPK@boomer.org with "RMSE" as the subject

Copyright 1995-2010 David W. A. Bourne (david@boomer.org)