Yesterday morning SPLUNK was working fine. I added some alerts to it and suddenly it all started going wrong. At one point I was also getting Maximum Historic Concurrent Searches as well. Finally I worked out how to increase that, but it didn't help and when I checked again I found it was maxsearches limit reached that was blocking everything. I have search answers.splunk.com and google but can find no other topic that has addressed this point. I'm on SPLUNK 6.2.2
Hi john_howley, this may help:
stackoverflow.com/questions/23763141/splunk-concurrent-seaches-user-vs-system
Thanks for the link Stephane, but that is for concurrent searches not maxsearches.. It seems the system has hit a brick wall independant of Concurrent searches.
which data were you indecing to the machine
Hi juvetm, the specific searches that are failing are dbqueries and NMON ones. The alerts I added yesterday were on file based index data held within SPLUNK - I have now disabled all those, but it still doesn't work. What I see if I look at the console is a load of dbqueries that have GETINFO prefix - those never clear so I end up having to kill -15 them.
can you try to limit this search in the configuration in
Limits.conf.spec
In limits.conf I have added under local to allow 2 concurrent searches per cpu and can see that has increased the max concurrent searches allowed, but as I said I can't find any cofig element for the overall maxsearches and I'm not 100% sure what you mean by 'limit this search'
try this
[subsearch]
maxout = integer number
So in lkmits.conf I have
[subsearch]
maxout = 10000
and
[join]
subsearch_maxout = 50000
the error is suggesting I have reached the maximum number of searches rather than the number of results returned from an individual search which these seem to suggest.
Are you suggesting increasing this number to allow more or reducing it so that results don't take up so many resources?
you should reduce it and let see what will happen waiting to hear from again
hi juvetm. I set it to 5000 from 10000 and restarted. The maxsearches limit is no longer appearing, but on further investigation I now see that it wasn't dbqueries that were getting this error it was nmon and alerts. I had disabled alerts yesterday to see if that would get me back to working again, but it didn't. Alerts are now running ok. I am still getting problems with dashboard SQL searches though - they are getting initiated but not completing and when I look at them on the console they all have GETINFO preceding the actual select element, but they never complete and I end up having to kill them. I can't see any evidence in the index=_internal for those queries at all now.