source count vs summary index count not match

[raventura]

Hi All,

have this dilemma where source counts does not match the count inserted in summary index. sample query that was used - 

Base search:

index=sample_index 
| rex mode=sed field=author"s/(\w|\d|[\D\W])/*/g" 
| eval raw_event=_raw 
| rex mode=sed field=raw_event"s/(:?author\=[\w|\d|\D\W]+)/author= *********/g" 
| fields user owner ip mac_address input_file dest_file log_name orig_time orig_sourcetype act category default message message_id raw_mac severity tag vendor product

then summary indexing is enabled. runs every 30 minutes that gets past 30 minutes.

Validation:

if base search is used, for example, it has a result of 100k events. when checked in summary index it has only 50% or less inserted.

Note that not all fields are present in all events. example for owner field, it has 3 Values, 17.377% of events.

Question:

does it summarize the fields being inserted to the summary index, where it drops the fields with null values? or is that the expected behavior from summary indexing?

Thanks!

[Richfez]

Could you repaste your search, only use the little code button?  The editor ate some characters.

Also, there's a note right there in the middle of the "considerations for summary index searches" that warns about using summary indexes for events and not summary statistics.

https://docs.splunk.com/Documentation/Splunk/8.0.6/Knowledge/Usesummaryindexing#Considerations_for_s...

It looks like your search, if the one you supplied is indeed the one populating your summary index, does not do any transforming commands and so falls into the category "getting them to work correctly can be tricky"

If the search you supplied is not what's feeding the summary index, can you please post that search too?

[raventura]

The only transformation did was to obfuscate some identified fields that needs masking. other than that, nothing else. re-insert all fields needed with the masked field/s to the summary index.

[Richfez]

Right, 'transforming command' means something very specific in Splunk.

https://docs.splunk.com/Splexicon:Transformingcommand

You don't have any. Since you don't have any, there's a special note in the docs I linked that you may have a hard time getting it to work right.

Think on it a second.  "Summary index" means "an index that summarizes".  Like, it summarizes error counts for the past 5 minutes, each 5 minutes.

What you've got may be better suited to routing and filtering?

https://docs.splunk.com/Documentation/Splunk/8.0.6/Forwarding/Routeandfilterdatad

So, stepping back just a bit.  What problem are you trying to solve?  What is the point of the summary index?