During the development of a project, we faced some problems concerning the output of correlation coefficient R^2, when using the Splunk Machine Learning Toolkit version 3.1.0.
About the results concerning the regression coefficient, when evaluating the performance of a fitted time series, as described here below, we obtain a negative R^2.
If we consider the R^2 as:
R^2=SSR/SST
where
SSR=sum_i (y_i-hat{y_i})^2 stands for Regression Sum of Squares and
SST=sum_i (y_i-overline{y})^2 for Total Sum of Squares,
it’s impossible to obtain negative values of it. But we are not sure how the algorithm calculates it.
We suspect that it’s being considered the adjusted R^2, since this measure may have a negative value in some particular situations. However, we are not sure of it and how do they extend the result to time series. Also, we haven’t found any information about it so far.
We hope that somebody can help us figure out what’s going on 🙂
All the best with your Splunk queries and work!
Hello @diasna1 - here is how we actually calculate the R^2 value as of MLTK 3.2.0:
| rename "$a$" as _actual, "$p$" as _predicted
| eventstats avg(_actual) as _avgActual
| eval _actualMinusAvg = _actual - _avgActual, _residual = _actual - _predicted
| stats sumsq(_actualMinusAvg) as _sumsqActualMinusAvg, sumsq(_residual) as _sumsqResidual, count(_residual) as _sampleCount
| eval rSquared = round(1 - _sumsqResidual / _sumsqActualMinusAvg, 4), RMSE = round(sqrt(_sumsqResidual / _sampleCount), 2)
| table rSquared RMSE
Please feel free to review the similar definition on wikipedia, and let me know if I made a mistake.
https://en.wikipedia.org/wiki/Coefficient_of_determination#Definitions
So to answer your question directly, R^2 can ben negative whenever SSR is greater than SST.
Additionally, you may find this cross validated question & answer relevant: https://stats.stackexchange.com/a/12991/78566
