Splunk Search

Why am I receiving 'maxsearches limit reached' in SPLUNK after adding some alerts?

john_howley
Path Finder

Yesterday morning SPLUNK was working fine. I added some alerts to it and suddenly it all started going wrong. At one point I was also getting Maximum Historic Concurrent Searches as well. Finally I worked out how to increase that, but it didn't help and when I checked again I found it was maxsearches limit reached that was blocking everything. I have search answers.splunk.com and google but can find no other topic that has addressed this point. I'm on SPLUNK 6.2.2

Tags (3)
0 Karma

stephane_cyrill
Builder

Hi john_howley, this may help:

stackoverflow.com/questions/23763141/splunk-concurrent-seaches-user-vs-system

0 Karma

john_howley
Path Finder

Thanks for the link Stephane, but that is for concurrent searches not maxsearches.. It seems the system has hit a brick wall independant of Concurrent searches.

0 Karma

juvetm
Communicator

which data were you indecing to the machine

0 Karma

john_howley
Path Finder

Hi juvetm, the specific searches that are failing are dbqueries and NMON ones. The alerts I added yesterday were on file based index data held within SPLUNK - I have now disabled all those, but it still doesn't work. What I see if I look at the console is a load of dbqueries that have GETINFO prefix - those never clear so I end up having to kill -15 them.

0 Karma

juvetm
Communicator

can you try to limit this search in the configuration in
Limits.conf.spec

0 Karma

john_howley
Path Finder

In limits.conf I have added under local to allow 2 concurrent searches per cpu and can see that has increased the max concurrent searches allowed, but as I said I can't find any cofig element for the overall maxsearches and I'm not 100% sure what you mean by 'limit this search'

0 Karma

juvetm
Communicator

try this
[subsearch]
maxout = integer number

0 Karma

john_howley
Path Finder

So in lkmits.conf I have
[subsearch]

maximum number of results to return from a subsearch

maxout = 10000
and
[join]
subsearch_maxout = 50000

the error is suggesting I have reached the maximum number of searches rather than the number of results returned from an individual search which these seem to suggest.
Are you suggesting increasing this number to allow more or reducing it so that results don't take up so many resources?

0 Karma

juvetm
Communicator

you should reduce it and let see what will happen waiting to hear from again

0 Karma

john_howley
Path Finder

hi juvetm. I set it to 5000 from 10000 and restarted. The maxsearches limit is no longer appearing, but on further investigation I now see that it wasn't dbqueries that were getting this error it was nmon and alerts. I had disabled alerts yesterday to see if that would get me back to working again, but it didn't. Alerts are now running ok. I am still getting problems with dashboard SQL searches though - they are getting initiated but not completing and when I look at them on the console they all have GETINFO preceding the actual select element, but they never complete and I end up having to kill them. I can't see any evidence in the index=_internal for those queries at all now.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...