Knowledge Management

Why are only 10,000 events making it into the summary index?

Dan
Splunk Employee
Splunk Employee

I'm having an issue with my summary index. I have a search which results in 48000+ events. I saved the search and enabled summary indexing on it. My issue is that only 10,000 events get put into the summary index each time the search runs.

Any idea what limit I need to change to get all my results in my summary index? The saved search is similar to this:

index="myidx" | rex field=source "data/(?<program_name>\w+)/(?<receive_date>\w+)___" | dedup source | stats list(program_name) as programName, list(receive_date) as receiveDate by source
1 Solution

Dan
Splunk Employee
Splunk Employee

There's by default a limit of 10,000 events that will get summary indexed from each run of a scheduled search. This default limit will actually be removed in most cases in the next maintenance release (4.1.2). For now, here's a couple of workarounds.

1) in savedsearches.conf for the app that your search belongs to, under the stanza for your scheduled searches, add dispatch.max_count=100000 (or whatever limit you want) ALSO, in etc/system/local/limits.conf (create it if it doesn't exist), under the [scheduler] stanza, set max_action_results=100000 (or a limit of your choosing).

OR

2) instead of setting enabling the summary indexing action of the saved searches, explicitly add a " | collect" to the end of your saved searches. This will change the search itself to directly populate your summary index. (instead of the default behavior, which is that the scheduler reads the result of your search and then populates the summary index from that result)

View solution in original post

Dan
Splunk Employee
Splunk Employee

There's by default a limit of 10,000 events that will get summary indexed from each run of a scheduled search. This default limit will actually be removed in most cases in the next maintenance release (4.1.2). For now, here's a couple of workarounds.

1) in savedsearches.conf for the app that your search belongs to, under the stanza for your scheduled searches, add dispatch.max_count=100000 (or whatever limit you want) ALSO, in etc/system/local/limits.conf (create it if it doesn't exist), under the [scheduler] stanza, set max_action_results=100000 (or a limit of your choosing).

OR

2) instead of setting enabling the summary indexing action of the saved searches, explicitly add a " | collect" to the end of your saved searches. This will change the search itself to directly populate your summary index. (instead of the default behavior, which is that the scheduler reads the result of your search and then populates the summary index from that result)

Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...