Getting Data In

Summary Index vs Report Acceleration?

tradecraft1914
Explorer

I have DNS logs from both Windows and Unix BIND. What I am trying to do is create a quick way for admins to query 90 days worth of these DNS logs to see if a particular domain name has ever been looked up. My thought was to do search of each index, do a "stats count by domain" and put the results potentially in a summary index. It was suggested that I look at Report Acceleration as well. I'm running into a couple issues.

  1. I cant save my query as accelerated. I do not see any checkbox in version 6.0. Here is the search: index=dnslog | stats count by domain_name

  2. It would be useful for the analysts to have something like a "first seen" date to go with the domains returned in the searches but I cannot figure out a way to do that.

Any help would be greatly appreciated.

1 Solution

ecambra_splunk
Splunk Employee
Splunk Employee

I would highly recommend using report acceleration, it is self repairing. Summary Indexing is entirely reliant on the scheduled search performing as expected.

Generally I only use summary indexing for summarized information that needs to be stored longer than a year.

Report Acceleration is still available in 6.0, and your search does qualify (uses a transforming command, stats). If you go to Settings > Searches & Reports > Your Search, do you not see the check box?

To calculate the first seen date you can use earliest() in your stats command. It would look like this...

| stats count earliest(eval(strftime(_time, "%H:%M:%S"))) AS "first seen" by domain_name

You can modify the time format to display month, date, year and all that other goodness.

View solution in original post

saravanan90
Contributor

+adding one more

Report acceleration saved in - $SPLUNK_HOME/var/lib/splunk/index1/summary whereas
Summary indexing will be saved in summary index($SPLUNK_HOME/var/lib/splunk/summarydb). Temporarily search results are stored in $SPLUNK_HOME/var/spool/splunk/_.stash

woodcock
Esteemed Legend

Splunk Report Acceleration:

  • Increases performance ~2-5x.
  • Automatically preforms backfill.
  • Late-arriving events are of NO concern.
  • Drilldown to raw data still works.
  • Speeds up a search by creating additional TSIDX mappings into the raw data.
  • Is incredibly robust and requires no housekeeping.
  • Is insensitive to TimeZone and the user who enabled the acceleration.
  • Has a minor "always-on" impact to Indexers and consumes a small amount of disk space but no license.
  • Does not require any conversion (just click the checkbox and you are done).
  • Replaces this question:

    Is there *possibly* a match for my data in this bucket?
    

    With this question:

    Is there definitely a match for my data in this bucket?

    Splunk Summary Index:

  • Astronomical increases in performance are possible (but, if you do it wrong, things can actually be slower).

  • Admin must manually preform backfill.

  • Late-arriving events are of GREAT concern

  • Drilldown to raw data no longer works (must do Advanced dashboard programming to force it to switch back to raw data).

  • Is somewhat fragile and requires reactive housekeeping (many things can cause SI data outages).

  • Is sensitive to TimeZone and the user who runs the "Populating Search".

  • Has a minor "every-hour" impact to Indexers and consumes a small amount of disk space but no license.

  • Speeds up searches by replacing huge amounts of raw data with much smaller amounts of aggregate ("summarized") data in the Summary Index.

  • Requires an Admin to reverse-engineer the desired end-usage of the data in order to construct an appropriate "Populating Search" to do this conversion.

  • Requires that all existing KO uses of the raw data be manually converted (e.g. saved searches, macros, dashboard panels) to pull data from the SI instead of raw data.

ecambra_splunk
Splunk Employee
Splunk Employee

I would highly recommend using report acceleration, it is self repairing. Summary Indexing is entirely reliant on the scheduled search performing as expected.

Generally I only use summary indexing for summarized information that needs to be stored longer than a year.

Report Acceleration is still available in 6.0, and your search does qualify (uses a transforming command, stats). If you go to Settings > Searches & Reports > Your Search, do you not see the check box?

To calculate the first seen date you can use earliest() in your stats command. It would look like this...

| stats count earliest(eval(strftime(_time, "%H:%M:%S"))) AS "first seen" by domain_name

You can modify the time format to display month, date, year and all that other goodness.

ecambra_splunk
Splunk Employee
Splunk Employee
0 Karma
Get Updates on the Splunk Community!

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...