Splunk Search

Streamed search execute failed because: Error in 'surrounding': Too many events (> 10000) in a single second.

splunk_zen
Builder

Even though I have overwritten what I believe is this limit in limits.conf,
btool is showing,

[show_source]
max_count = 50000
distributed_search_limit = 30000
distributed = true

The error message displays 10k rather than 50k.
Is this a bug as in the parameter is not being respected, or a bug as in the message not displaying the value Splunk is enforcing ?

Any recommendation on how to allow to check the source for

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

1st, know your limits:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Limitsconf

limits.conf settings and DISTRIBUTED SEARCH
Unlike most settings which affect searches, limits.conf settings are not
provided by the search head to be used by the search peers.
This means that if
you need to alter search-affecting limits in a distributed environment, typically
you will need to modify these settings on the relevant peers and search head for
consistent results.

2nd, tell us your architecture. If you only have 1 server, my answer above is null and void.

3rd, as mentioned there is a configuration file precedence issue possibly. See the following:
http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Wheretofindtheconfigurationfiles

View solution in original post

jkat54
SplunkTrust
SplunkTrust

1st, know your limits:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Limitsconf

limits.conf settings and DISTRIBUTED SEARCH
Unlike most settings which affect searches, limits.conf settings are not
provided by the search head to be used by the search peers.
This means that if
you need to alter search-affecting limits in a distributed environment, typically
you will need to modify these settings on the relevant peers and search head for
consistent results.

2nd, tell us your architecture. If you only have 1 server, my answer above is null and void.

3rd, as mentioned there is a configuration file precedence issue possibly. See the following:
http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Wheretofindtheconfigurationfiles

splunk_zen
Builder

Thanks jkat54.
That makes sense, will push these changes to the Indexer cluster then.
There's no configuration file precedence issue as confirmed by btool, set this up in a specific App to exclusively target limits.conf (thus taking predence over system/{default,local})

jkat54
SplunkTrust
SplunkTrust

number 1 should help you then! Thanks for marking the answer.

0 Karma

renjith_nair
Legend

Try to do that in local/limits.conf and restart splunk after that if not done already

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Event Series: Splunk Observability Metrics Cost Optimization

Balancing Scale and Spend: Gaining Control Over High-Volume Metrics in Splunk Observability Cloud As ...

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...

Deep insights, no barriers: Splunk Observability Cloud Free Edition

As software delivery cycles continue to accelerate, observability shouldn’t be a luxury — it should be a ...