Knowledge Management

Summary indexing

Communicator

Hi

According to this page
http://docs.splunk.com/Documentation/Splunk/6.0.3/Knowledge/Usesummaryindexing

stuff that is sent to a summary index has its fields renamed, the sourcetyp gets renamed to stash and the orig. sourcetype should be called origsourcetype.
Using version 6.0.3 I get the sourcetype field renamed for stuff sent to the summary index but there is no orig
sourcetype field.

Are there any special requisites to get those fields ? Anything I need to do or are not allowed to do ?

0 Karma

Communicator

What I've learned with summary indexing and transactions is there are some inconsistencies in the collection of the data ... It turned out I have to do the transaction in the summary index, not before because my main concern with the duration ... I am collecting VPN logs. so I ended up using a join command to gather my events, used collect to push them into the summary index and then I run the transaction command using the startswith/endswith and eval on the fields.. that solved a big part of my problem.

| join sessionid [ search agency=sales] .... this gathers all my relevant events with a sessionid field and narrows the grouping down by doing a search for the field agency=sales

I also found out that I had to create all new field extractions using the sourcetype=stash because that is what the summary index uses ... other summary indexes will also use the same sourcetype=stash too, although I think you can force a different name ... luckily I didn't need to figure that out.

Maybe this is useful, maybe not ... but thought I would try ... Oh, and remember to make sure your in smartmode instead of fast mode for field discovery

0 Karma

Splunk Employee
Splunk Employee

The sourcetype of a summary event will always be "stash_new".
And a field "orig_sourcetype" will be created only if such a field is necessary in your summary generation search.

example :
<mysearch> | sistats count by sourcetype
will silently replace sourcetype by "origsourcetype" and revert it back when you retrieve the summarized result with the symmetrical search
`index=summary search
name=mysummarysearch | stats count by sourcetype`

However a summary populating search like
<mysearch> | sistats count by host
will not create the field "orig_summary" because there is no need for it in the summarization.

0 Karma

Splunk Employee
Splunk Employee

the sourcetype is the only exception....

0 Karma

Splunk Employee
Splunk Employee

"fields created by the eval statements", it depends if the fields are renamed BEFORE or AFTER the summary command.

example :

index=* | rename host AS myhost | sistats count by myhost will be summarized with the field myhost

while

index=* | sistats count by host | rename host AS myhost will be summarized at the "sistats" level, therefore any renaming will not be saved, and the field will be "host"

this is the same for renamed commands in the summary command.
index=* | sistats count AS mycount by host the renaming of the count field will not be saved either.

0 Karma

Communicator

Why doesn't splunk keep / rename the fields created by the eval statements ? Those are the fields that I would need as fields in the summary automatically ..

0 Karma

Communicator

The search itself works and provides the fields that I want / need but saving the results to a summary index will neither get me the name / newname fields nor the origsourcetype field that I think should be automatically created.

index=someindex sourcetype=auditd | sort msg desc | transaction name maxspan=2s startswith=(syscall=open AND a1=c1) endswith=(syscall=unlink) keeporphans=true | where isnull(eventcount) | eval name=if(isnull(origname),name,origname) | eval newname=if(isnull(newname),"",newname)

0 Karma

Legend

Please show the search that you are using to create entries in the summary index... thanks!

0 Karma

Communicator

I expected the sourcetype field that splunk automatically generates for every event to be renamed to orig_sourcetype acc. to the mentioned webpage but that does not happen.

My query indeed DOES generate some fields but with unsed names and they are dumped alltogether anyway and are therefor NOT in the summary index 😞

0 Karma

SplunkTrust
SplunkTrust

Does your query, result of which you're saving/sending to summary index, generates field sourcetype? During summary indexing following fields are being overwritten and if these fields are present in the summary index query result, they will get renamed to orig_fieldname.

host

source
sourcetype
index

0 Karma