topic Re: Summary indexing in Knowledge Management

Summary indexing

flo_cognosec — Mon, 28 Sep 2020 16:46:55 GMT

According to this page
http://docs.splunk.com/Documentation/Splunk/6.0.3/Knowledge/Usesummaryindexing

stuff that is sent to a summary index has its fields renamed, the sourcetyp gets renamed to stash and the orig. sourcetype should be called orig_sourcetype.
Using version 6.0.3 I get the sourcetype field renamed for stuff sent to the summary index but there is no orig_sourcetype field.

Are there any special requisites to get those fields ? Anything I need to do or are not allowed to do ?

Re: Summary indexing

somesoni2 — Tue, 03 Jun 2014 13:46:46 GMT

Does your query, result of which you're saving/sending to summary index, generates field sourcetype? During summary indexing following fields are being overwritten and if these fields are present in the summary index query result, they will get renamed to orig_fieldname.

host

source
sourcetype
index

Re: Summary indexing

flo_cognosec — Tue, 03 Jun 2014 13:53:39 GMT

I expected the sourcetype field that splunk automatically generates for every event to be renamed to orig_sourcetype acc. to the mentioned webpage but that does not happen.

My query indeed DOES generate some fields but with unsed names and they are dumped alltogether anyway and are therefor NOT in the summary index 😞

Re: Summary indexing

lguinn2 — Tue, 03 Jun 2014 20:47:09 GMT

Please show the search that you are using to create entries in the summary index... thanks!

Re: Summary indexing

flo_cognosec — Mon, 28 Sep 2020 16:47:11 GMT

The search itself works and provides the fields that I want / need but saving the results to a summary index will neither get me the name / new_name fields nor the orig_sourcetype field that I think should be automatically created.

index=some_index sourcetype=auditd | sort msg desc | transaction name maxspan=2s startswith=(syscall=open AND a1=c1) endswith=(syscall=unlink) keeporphans=true | where isnull(eventcount) | eval name=if(isnull(orig_name),name,orig_name) | eval new_name=if(isnull(new_name),"",new_name)

Re: Summary indexing

yannK — Mon, 28 Sep 2020 16:58:07 GMT

The sourcetype of a summary event will always be "stash_new".
And a field "orig_sourcetype" will be created only if such a field is necessary in your summary generation search.

example :
<mysearch> | sistats count by sourcetype
will silently replace sourcetype by "orig_sourcetype" and revert it back when you retrieve the summarized result with the symmetrical search
index=summary search_name=mysummarysearch | stats count by sourcetype

However a summary populating search like
<mysearch> | sistats count by host
will not create the field "orig_summary" because there is no need for it in the summarization.

Re: Summary indexing

flo_cognosec — Tue, 01 Jul 2014 07:27:08 GMT

Why doesn't splunk keep / rename the fields created by the eval statements ? Those are the fields that I would need as fields in the summary automatically ..

Re: Summary indexing

yannK — Tue, 01 Jul 2014 16:31:06 GMT

"fields created by the eval statements", it depends if the fields are renamed BEFORE or AFTER the summary command.

example :

index=* | rename host AS myhost | sistats count by myhost will be summarized with the field myhost

while

index=* | sistats count by host | rename host AS myhost will be summarized at the "sistats" level, therefore any renaming will not be saved, and the field will be "host"

this is the same for renamed commands in the summary command.
index=* | sistats count AS mycount by host the renaming of the count field will not be saved either.

Re: Summary indexing

yannK — Tue, 01 Jul 2014 16:32:45 GMT

the sourcetype is the only exception....

Re: Summary indexing

rewritex — Tue, 29 Sep 2020 09:51:47 GMT

What I've learned with summary indexing and transactions is there are some inconsistencies in the collection of the data ... It turned out I have to do the transaction in the summary index, not before because my main concern with the duration ... I am collecting VPN logs. so I ended up using a join command to gather my events, used collect to push them into the summary index and then I run the transaction command using the startswith/endswith and eval on the fields.. that solved a big part of my problem.

| join session_id [ search agency=sales] .... this gathers all my relevant events with a session_id field and narrows the grouping down by doing a search for the field agency=sales

I also found out that I had to create all new field extractions using the sourcetype=stash because that is what the summary index uses ... other summary indexes will also use the same sourcetype=stash too, although I think you can force a different name ... luckily I didn't need to figure that out.

Maybe this is useful, maybe not ... but thought I would try ... Oh, and remember to make sure your in smartmode instead of fast mode for field discovery