What is the best and most economical method for keeping a continuously maintained reference of historical values?

[DrFedtke]

Hi all,

We want to compare "today" values in real-time with some aggregatedvalues of yesterday ("day -1"), "day -2", "week mean value", ... etc.

Instead of keeping all the old data (causing volume) to calculate all these reference values anew each day, we want to determine day-related totals, mean values, etc. each day at 1am, and store these values in a kind of "reference table". This allows us to delete old data. Deviations of the real-time monitoring are calculated by referring to that reference table.

What is the best way to realize such a mechanism in Splunk?

Is a lookup table the best choice for keeping the "day(-1)",
... histories? or is there any better method?

Thanks for any tip, link, or sample code.

best, and thanks to all
Caspar

[Richfez]

I second bmacias84's thought, a summary index sounds like exactly what you need. For a very small, isolated set a lookup table would work fine, but what I've found is that as soon as you implement that, you'll realize you also want per hour history, per week, maybe per minute... and it snowballs out of easy manageability.

In addition to his link for the official documentation on Configuring Summary Indexes, I'd recommend an additional resource: Go to the .Conf 2013 session page here and watch the breakout session "Automating Operational Intelligence: Stats and Summary Indexes" by Jesse Trucks. It is a great run-through of creating one.

[somesoni2]

If the no of aggregated values for each day is very small, you can use lookup table for faster response. You'd have to create a scheduled search to run daily and append yesterday's aggregated to the lookup table.

[bmacias84]

using Summary indexing with sistats or sitimechart.

http://docs.splunk.com/Documentation/Splunk/6.2.4/Knowledge/Configuresummaryindexes