What are the differences between the various features named "summary" in Splunk?

[mataharry]

They are many features using objects named "summary", this is confusing, please clarify.

what are the differences between all those paths ?

$SPLUNK_HOME/var/lib/splunk/summary/db
$SPLUNK_HOME/var/lib/splunk/defaultdb/summary 
$SPLUNK_HOME/var/lib/splunk/defaultdb/datamodel_summary

In savedsearches, what means auto_summarize and alert.action=summary

[yannK · Accepted solution]

To clarify there are 3 features named "summary" in splunk, :

A - Summary indexing : classic since splunk 4.*

• populated by scheduled searches, can use special "si*" stats commands (use the parameter alert.action=summary in savedsearches.conf)
• results are saved in the spooler and reindexed with the sourcetype stash_new
• stored in an index of your choice.
• an index named "summary" is shipped with splunk by default ($SPLUNK_HOME/var/lib/splunk/summary/db)
• the results have to be retrieved with special searches syntax
• docs : http://docs.splunk.com/Documentation/Splunk/6.1.3/Knowledge/Usesummaryindexing

B - Report acceleration : introduced on splunk 5.*

• Some searches can be accelerated, (use the parameter auto_summarize=true in savedsearches.conf)
• for each index a folder named "summary" is created at the same level than the homePath. ( example $SPLUNK_HOME/var/lib/splunk/defaultdb/summary)
• The acceleration is transparent.
• docs http://docs.splunk.com/Documentation/Splunk/6.1.3/Knowledge/Aboutsummaryindexing#Report_acceleration

C - Data model acceleration : introduced on splunk 6.*

• the data models can be accelerated, (use the parameter acceleration=true in datamodels.conf)
• the results are stored per index in the folder named "datamodel_summary" at the same level than the homePath. (example $SPLUNK_HOME/var/lib/splunk/defaultdb/datamodel_summary )
• The acceleration rely on tstats commands and is transparent.
• docs : http://docs.splunk.com/Documentation/Splunk/6.1.3/Knowledge/Aboutsummaryindexing#Data_model_accelera...

Remark : none of those features counts on your license usage, but they can add some extra search load to generate the summarized data.

[jaredlaney]

@yannK - Is there anyway you could explain these in more of a conceptual vs. a mechanical way?

[jaredlaney]

For example, maybe explain if one is more like an additional index vs. one being a cache? Maybe some good cases of when to use one vs. when to use another.

[yannK]

Those methods A and B are not supposed to complete each others they are just 2 ways to achieve the same thing.

A - The "Summary indexing" is like generating events in a new index.
It's is perfect to generate a new set of pre-calculated data, and keep it for a longer retention.
example : having millions of web acccess logs in an index with a short retention, and every day summarize them as a number of hit per day, store in an dedicated index with a long retention. At the end you will only keep this information.
The only difficulty is if a scheduled search is skipped, you may have a gap to backfill

B - Report acceleration is for searches only, it precaculate them for you.
Example : having a long statistical search over a long period to populate a dashboard. Accelerate it to run all the time in the background , and load faster.

C - Data model acceleration is only usefull if you already have a datamodel. They are usually heavier to run, so accelerating them will help.
Example : the Common Information Model (CIM) comes with many datamodels, once the volume is large, the searches are slower. When the acceleration is turned on (depending of thebackfill range), it will be faster for the recent days.

[mic]

yannK mentioned the following

an index named "summary" is shipped with splunk by default ($SPLUNK_HOME/var/lib/splunk/summary/db)

I believe it's $SPLUNK_HOME/var/lib/splunk/summarydb, not $SPLUNK_HOME/var/lib/splunk/summary/db. Notice that there is no backslash between "summary" and "db".