How do scheduled searches work?

[perfecto25]

General question about how scheduling searching behaves,

we have a 3 node SH cluster and couple of indexers, and the SH cluster has 5-10 custom inhouse apps that do a ton of searches, very heavy on mem usage

we are seeing lots of "out of memory" errors on some of the SH nodes and some of the indexers, and the only thing that I can think of that can be eating up this much memory is our searches

When a SH starts a scheduled search, does it impact Indexer's performance in any way or is all the memory usage only on the Search Head itself? How does the indexer and SH break down the search during runtime?

I'm thinking of adding search limits (in terms of how much memory each search can use) using limits.conf on each search head.

Thanks

[woodcock]

Get all the RAM that you can for your Search Heads, then for your Indexers. Max them out. The cost is low and the benefit is tremendous. Also, upgrade to 7.1.2 the day that it comes out. There are MAJOR memory leaks in all 7.* versions, but ESPECIALLY 7.1.*.

[DalJeanis]

When talking about "out of memory" errors, it would be good to tell us what version of splunk you are on. There have been memory leak issues in the 7.X versions, so if you're on one of those advanced versions, it's not necessarily your searches that are the issue.

Please join the splunk slack channel, and chat in the #general sub-channel in order to do a quick triage on your issue, and we'll go from there.

[perfecto25]

thanks for the heads up, I applied to join the slack channel

is there a document that details these issues? Dont see anything about mem leaks in 7.1 release notes/known issues

[perfecto25]

our SH cluster is running 7.1.1

[renjith_nair]

Hi @perfecto25,

Regardless of whether the search is a scheduled search or on demand search, indexers are involved since your data is stored in indexers. Your search is sent to the indexers and look for data based on your search criteria and pulls results. The retrieved data is further processed based on the subsequent search commands.

Have a look at this .conf presentation to have a better understanding How search works

Also since your environment is clustered, please refer to How search works in an indexer cluster

Also The anatomy of a search will give you more information

[perfecto25]

Thank you! Will read through this.