Splunk Search

source count vs summary index count not match

raventura
Observer

Hi All,

have this dilemma where source counts does not match the count inserted in summary index. sample query that was used - 

Base search:

index=sample_index 
| rex mode=sed field=author"s/(\w|\d|[\D\W])/*/g" 
| eval raw_event=_raw 
| rex mode=sed field=raw_event"s/(:?author\=[\w|\d|\D\W]+)/author= *********/g" 
| fields user owner ip mac_address input_file dest_file log_name orig_time orig_sourcetype act category default message message_id raw_mac severity tag vendor product

then summary indexing is enabled. runs every 30 minutes that gets past 30 minutes.

Validation:

if base search is used, for example, it has a result of 100k events. when checked in summary index it has only 50% or less inserted.

Note that not all fields are present in all events. example for owner field, it has 3 Values, 17.377% of events.

 

Question:

does it summarize the fields being inserted to the summary index, where it drops the fields with null values? or is that the expected behavior from summary indexing?

 

Thanks!

 

 

Labels (3)
Tags (1)
0 Karma

Richfez
SplunkTrust
SplunkTrust

Could you repaste your search, only use the little code button?  The editor ate some characters.

Also, there's a note right there in the middle of the "considerations for summary index searches" that warns about using summary indexes for events and not summary statistics.

https://docs.splunk.com/Documentation/Splunk/8.0.6/Knowledge/Usesummaryindexing#Considerations_for_s...

It looks like your search, if the one you supplied is indeed the one populating your summary index, does not do any transforming commands and so falls into the category "getting them to work correctly can be tricky"

If the search you supplied is not what's feeding the summary index, can you please post that search too?

0 Karma

raventura
Observer

The only transformation did was to obfuscate some identified fields that needs masking. other than that, nothing else. re-insert all fields needed with the masked field/s to the summary index.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Right, 'transforming command' means something very specific in Splunk.

https://docs.splunk.com/Splexicon:Transformingcommand

You don't have any. Since you don't have any, there's a special note in the docs I linked that you may have a hard time getting it to work right.

Think on it a second.  "Summary index" means "an index that summarizes".  Like, it summarizes error counts for the past 5 minutes, each 5 minutes.

What you've got may be better suited to routing and filtering?

https://docs.splunk.com/Documentation/Splunk/8.0.6/Forwarding/Routeandfilterdatad

So, stepping back just a bit.  What problem are you trying to solve?  What is the point of the summary index?

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...