Splunk Search

Streamed search execute failed because: Error in 'surrounding': Too many events (> 10000) in a single second.

splunk_zen
Builder

Even though I have overwritten what I believe is this limit in limits.conf,
btool is showing,

[show_source]
max_count = 50000
distributed_search_limit = 30000
distributed = true

The error message displays 10k rather than 50k.
Is this a bug as in the parameter is not being respected, or a bug as in the message not displaying the value Splunk is enforcing ?

Any recommendation on how to allow to check the source for

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

1st, know your limits:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Limitsconf

limits.conf settings and DISTRIBUTED SEARCH
Unlike most settings which affect searches, limits.conf settings are not
provided by the search head to be used by the search peers.
This means that if
you need to alter search-affecting limits in a distributed environment, typically
you will need to modify these settings on the relevant peers and search head for
consistent results.

2nd, tell us your architecture. If you only have 1 server, my answer above is null and void.

3rd, as mentioned there is a configuration file precedence issue possibly. See the following:
http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Wheretofindtheconfigurationfiles

View solution in original post

jkat54
SplunkTrust
SplunkTrust

1st, know your limits:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Limitsconf

limits.conf settings and DISTRIBUTED SEARCH
Unlike most settings which affect searches, limits.conf settings are not
provided by the search head to be used by the search peers.
This means that if
you need to alter search-affecting limits in a distributed environment, typically
you will need to modify these settings on the relevant peers and search head for
consistent results.

2nd, tell us your architecture. If you only have 1 server, my answer above is null and void.

3rd, as mentioned there is a configuration file precedence issue possibly. See the following:
http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Wheretofindtheconfigurationfiles

splunk_zen
Builder

Thanks jkat54.
That makes sense, will push these changes to the Indexer cluster then.
There's no configuration file precedence issue as confirmed by btool, set this up in a specific App to exclusively target limits.conf (thus taking predence over system/{default,local})

jkat54
SplunkTrust
SplunkTrust

number 1 should help you then! Thanks for marking the answer.

0 Karma

renjith_nair
Legend

Try to do that in local/limits.conf and restart splunk after that if not done already

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Get Agentic with Splunk Lantern: Connect to Cisco Cloud Control, Transform ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

July Community Events: Master ITSI 5.0 & Automate Splunk

Struggling with alert fatigue or feeling like you're spending more time on infrastructure maintenance than ...

New Release of Federated Search: Bringing Splunk Analytics to More of Your Data

Organizations today are generating more data than ever and storing it across cloud object stores, data lakes, ...