- Splunk Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- Re: How do scheduled searches work?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How do scheduled searches work?
General question about how scheduling searching behaves,
we have a 3 node SH cluster and couple of indexers, and the SH cluster has 5-10 custom inhouse apps that do a ton of searches, very heavy on mem usage
we are seeing lots of "out of memory" errors on some of the SH nodes and some of the indexers, and the only thing that I can think of that can be eating up this much memory is our searches
When a SH starts a scheduled search, does it impact Indexer's performance in any way or is all the memory usage only on the Search Head itself? How does the indexer and SH break down the search during runtime?
I'm thinking of adding search limits (in terms of how much memory each search can use) using limits.conf on each search head.
Thanks
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Get all the RAM that you can for your Search Heads, then for your Indexers. Max them out. The cost is low and the benefit is tremendous. Also, upgrade to 7.1.2 the day that it comes out. There are MAJOR memory leaks in all 7.* versions, but ESPECIALLY 7.1.*.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When talking about "out of memory" errors, it would be good to tell us what version of splunk you are on. There have been memory leak issues in the 7.X versions, so if you're on one of those advanced versions, it's not necessarily your searches that are the issue.
Please join the splunk slack channel, and chat in the #general sub-channel in order to do a quick triage on your issue, and we'll go from there.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thanks for the heads up, I applied to join the slack channel
is there a document that details these issues? Dont see anything about mem leaks in 7.1 release notes/known issues
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
our SH cluster is running 7.1.1
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @perfecto25,
Regardless of whether the search is a scheduled search or on demand search, indexers are involved since your data is stored in indexers. Your search is sent to the indexers and look for data based on your search criteria and pulls results. The retrieved data is further processed based on the subsequent search commands.
Have a look at this .conf presentation to have a better understanding How search works
Also since your environment is clustered, please refer to How search works in an indexer cluster
Also The anatomy of a search will give you more information
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank you! Will read through this.
Announcing Scheduled Export GA for Dashboard Studio
Extending Observability Content to Splunk Cloud
More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!
