Getting Data In
Highlighted

Summary Index vs Report Acceleration?

Explorer

I have DNS logs from both Windows and Unix BIND. What I am trying to do is create a quick way for admins to query 90 days worth of these DNS logs to see if a particular domain name has ever been looked up. My thought was to do search of each index, do a "stats count by domain" and put the results potentially in a summary index. It was suggested that I look at Report Acceleration as well. I'm running into a couple issues.

  1. I cant save my query as accelerated. I do not see any checkbox in version 6.0. Here is the search: index=dnslog | stats count by domain_name

  2. It would be useful for the analysts to have something like a "first seen" date to go with the domains returned in the searches but I cannot figure out a way to do that.

Any help would be greatly appreciated.

Highlighted

Re: Summary Index vs Report Acceleration?

Splunk Employee
Splunk Employee

I would highly recommend using report acceleration, it is self repairing. Summary Indexing is entirely reliant on the scheduled search performing as expected.

Generally I only use summary indexing for summarized information that needs to be stored longer than a year.

Report Acceleration is still available in 6.0, and your search does qualify (uses a transforming command, stats). If you go to Settings > Searches & Reports > Your Search, do you not see the check box?

To calculate the first seen date you can use earliest() in your stats command. It would look like this...

| stats count earliest(eval(strftime(time, "%H:%M:%S"))) AS "first seen" by domainname

You can modify the time format to display month, date, year and all that other goodness.

View solution in original post

Highlighted

Re: Summary Index vs Report Acceleration?

Splunk Employee
Splunk Employee
0 Karma
Highlighted

Re: Summary Index vs Report Acceleration?

Esteemed Legend

Splunk Report Acceleration:

  • Increases performance ~2-5x.
  • Automatically preforms backfill.
  • Late-arriving events are of NO concern.
  • Drilldown to raw data still works.
  • Speeds up a search by creating additional TSIDX mappings into the raw data.
  • Is incredibly robust and requires no housekeeping.
  • Is insensitive to TimeZone and the user who enabled the acceleration.
  • Has a minor "always-on" impact to Indexers and consumes a small amount of disk space but no license.
  • Does not require any conversion (just click the checkbox and you are done).
  • Replaces this question:

    Is there *possibly* a match for my data in this bucket?
    

    With this question:

    Is there definitely a match for my data in this bucket?

    Splunk Summary Index:

  • Astronomical increases in performance are possible (but, if you do it wrong, things can actually be slower).

  • Admin must manually preform backfill.

  • Late-arriving events are of GREAT concern

  • Drilldown to raw data no longer works (must do Advanced dashboard programming to force it to switch back to raw data).

  • Is somewhat fragile and requires reactive housekeeping (many things can cause SI data outages).

  • Is sensitive to TimeZone and the user who runs the "Populating Search".

  • Has a minor "every-hour" impact to Indexers and consumes a small amount of disk space but no license.

  • Speeds up searches by replacing huge amounts of raw data with much smaller amounts of aggregate ("summarized") data in the Summary Index.

  • Requires an Admin to reverse-engineer the desired end-usage of the data in order to construct an appropriate "Populating Search" to do this conversion.

  • Requires that all existing KO uses of the raw data be manually converted (e.g. saved searches, macros, dashboard panels) to pull data from the SI instead of raw data.

Highlighted

Re: Summary Index vs Report Acceleration?

Path Finder

+adding one more

Report acceleration saved in - $SPLUNKHOME/var/lib/splunk/index1/summary whereas
Summary indexing will be saved in summary index($SPLUNK
HOME/var/lib/splunk/summarydb). Temporarily search results are stored in $SPLUNKHOME/var/spool/splunk/.stash