Knowledge Management

Why are only 10,000 events making it into the summary index?

Dan
Splunk Employee
Splunk Employee

I'm having an issue with my summary index. I have a search which results in 48000+ events. I saved the search and enabled summary indexing on it. My issue is that only 10,000 events get put into the summary index each time the search runs.

Any idea what limit I need to change to get all my results in my summary index? The saved search is similar to this:

index="myidx" | rex field=source "data/(?<program_name>\w+)/(?<receive_date>\w+)___" | dedup source | stats list(program_name) as programName, list(receive_date) as receiveDate by source
1 Solution

Dan
Splunk Employee
Splunk Employee

There's by default a limit of 10,000 events that will get summary indexed from each run of a scheduled search. This default limit will actually be removed in most cases in the next maintenance release (4.1.2). For now, here's a couple of workarounds.

1) in savedsearches.conf for the app that your search belongs to, under the stanza for your scheduled searches, add dispatch.max_count=100000 (or whatever limit you want) ALSO, in etc/system/local/limits.conf (create it if it doesn't exist), under the [scheduler] stanza, set max_action_results=100000 (or a limit of your choosing).

OR

2) instead of setting enabling the summary indexing action of the saved searches, explicitly add a " | collect" to the end of your saved searches. This will change the search itself to directly populate your summary index. (instead of the default behavior, which is that the scheduler reads the result of your search and then populates the summary index from that result)

View solution in original post

Dan
Splunk Employee
Splunk Employee

There's by default a limit of 10,000 events that will get summary indexed from each run of a scheduled search. This default limit will actually be removed in most cases in the next maintenance release (4.1.2). For now, here's a couple of workarounds.

1) in savedsearches.conf for the app that your search belongs to, under the stanza for your scheduled searches, add dispatch.max_count=100000 (or whatever limit you want) ALSO, in etc/system/local/limits.conf (create it if it doesn't exist), under the [scheduler] stanza, set max_action_results=100000 (or a limit of your choosing).

OR

2) instead of setting enabling the summary indexing action of the saved searches, explicitly add a " | collect" to the end of your saved searches. This will change the search itself to directly populate your summary index. (instead of the default behavior, which is that the scheduler reads the result of your search and then populates the summary index from that result)

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...