topic Re: is it right to use summary index to save non-statistical data? in Knowledge Management

is it right to use summary index to save non-statistical data?

RiccardoV — Thu, 30 Jan 2014 15:58:31 GMT

Hi,
I'm using summary index, but I am not sure if I'm doing it right.
I have several searches that save data into my summary index. Some of them are saving statistical data, ie. how many events for each category I found.
But I need to speed up some different queries, where I need to display a table with many fields, ie. md5 - process - timestamp - category - etc.

Is it correct to run a scheduled query that saves data in summary index in that way?

Thanks!

edit: here is my search

sourcetype="tcp-raw" type=WSAEXEC OR type=WSAPROCESS
| eval ...
| search ...
| stats count ...
| stats dc(det) AS ...
| eval Category = ...
| table field1, field2, field3....

I did some changes, of couse 🙂

I retrieve the data in my dashboard in this way:

index="summary" 
| dedup field1, field2
| stats count by field1, field2
| sort -count
| head 10
| fields field1, field2, ....

thanks, again 🙂

edit #2:

here is my new search, only with streamable commands:

sourcetype="tcp-raw" type=WSAEXEC OR type=WSAPROCESS
| eval ...
| search ...
| stats count ...
| stats dc(det) AS ...
| eval Category = ...
| table field1, field2, field3....

I activated report acceleration with "1 month" summary range. Do I Have to leave empty start time and finish time values in time range?

How can I retrieve this "accelerated" data now? Just doing the same exact search I accelerated?

thanks!

Re: is it right to use summary index to save non-statistical data?

lguinn2 — Thu, 30 Jan 2014 17:01:59 GMT

If you want to speed up searches, but not save statistical data, use report acceleration instead of summary indexing. That said, there are rules about which searches can be accelerated.

Can you show us the actual search that you want to run?

Re: is it right to use summary index to save non-statistical data?

RiccardoV — Fri, 31 Jan 2014 09:04:11 GMT

thanks @lguinn for your answer. I've just updated my question with the searches!

Re: is it right to use summary index to save non-statistical data?

lguinn2 — Fri, 31 Jan 2014 22:57:17 GMT

A search that contains a transaction command can't be accelerated. But I still think that report acceleration might be a better way to do this. Why are you using the transaction command? I think this might be optimized - quite a lot, actually.

Re: is it right to use summary index to save non-statistical data?

RiccardoV — Mon, 03 Feb 2014 09:49:25 GMT

thanks again for your reply @lguinn!
I modified my search and now I'm 100% transaction-free 🙂

I updated the search in first post!

Re: is it right to use summary index to save non-statistical data?

RiccardoV — Mon, 03 Feb 2014 15:40:11 GMT

Edit #2: I updated again my inizial search with only streamable commands and I accelerated my search..please have a look!

Re: is it right to use summary index to save non-statistical data?

lguinn2 — Tue, 11 Feb 2014 22:43:17 GMT

You said "I activated report acceleration with "1 month" summary range. Do I Have to leave empty start time and finish time values in time range?

How can I retrieve this "accelerated" data now? Just doing the same exact search I accelerated?"

Answers:

First, you do not need to leave the start time and finish times empty in the time range. Splunk figures out how to accelerate date in the 1 month range automatically, regardless of the start and finish times.

Second, whenever you run this search, it will be accelerated. You don't need to do anything more; there are no special steps. In fact, if you run a similar search and Splunk can leverage the underlying acceleration summary, it will!

Finally, you shouldn't need to do any maintenance on the acceleration summary; Splunk will keep it valid and up to date.

Re: is it right to use summary index to save non-statistical data?

RiccardoV — Wed, 12 Feb 2014 08:36:33 GMT

thanks again for you answers @lguinn!