My schedule jobs are skipping all the time and getting following reasons:
ES is installed in 32 CPU Cores and 64GB RAM machine.
Lots of Jobs are running under ESS admin account.
For now, I didn't change any of limits.conf and Authorize.conf.
Can anyone help with the reason for above messages and how to set limits.conf and Authorize.conf with the calculation.
Any Help would be Appreciated!
Thanks in advance!
Check your limits.conf. Its recommended to change the limits on your ES SH for the scheduler:
[scheduler] auto_summary_perc = 100 max_searches_perc = 75
Additionally, you can check the monitoring console for searches that are taking too long to run. Since there are several moving parts in ES (correlation search, data model acceleration searches, etc) slow searches can have an impact on your system. Also take look at you indexer layer since it can also contribute to search slowness.
If you post more details (infrastructure sizing, # correlation searches, accelerated data models, etc. ) maybe we can help more.
sizing is 25 GB per day. I have 2 Indexers (Cluster) 16 core and 32 GB RAM, 2 SH (cluster) 16 core and 32 GB RAM, 1 SH standalone for ES 32 core and 64 GB RAM.
All are in windows servers 2016
13 data models are accelerated and 16 correlation searches are enabled..
and Just want to know any calculations to keep limits.conf? If yes, Please let me know the formula and calculation for my Environment.
Assuming you want to try and improve situation till you add additional Indexers, you could do the following:
So in total you have more search heads than you have indexers? That might be a problem... Also you're missing CM/LM/Deployer in your list of instances
whatever you're doing in the in the other SHs will also have impact into the IDX cluster performance wise and must be accounted for. Im guessing you might have warnings the the non-ES SHs complaining about performance?
Ad-Hoc search will have priority over other kind of searches (Scheduled, DMA, etc) so if you have a small index tier with several SHs searching there its a possibility that you'll end up with deferred/skipped searches. Consider increasing the number of indexers in your cluster.
Check splunk's recommended architectures:
More details about search priority from docs:
For more details on the scheduler check this .conf presentation:https://conf.splunk.com/files/2017/slides/making-the-most-of-the-splunk-scheduler.pdf