Need a help with skipped saved searches and help with calculation in limits

[satyaallaparthi]

Hello,

My schedule jobs are skipping all the time and getting following reasons:

1. The maximum number of concurrent auto-summarization searches on this instance has been reached
2. The maximum number of concurrent historical scheduled searches on this instance has been reached
3. The maximum number of concurrent running jobs for this historical scheduled search on this instance has been reached

ES is installed in 32 CPU Cores and 64GB RAM machine.

Lots of Jobs are running under ESS admin account.

For now, I didn't change any of limits.conf and Authorize.conf.

Can anyone help with the reason for above messages and how to set limits.conf and Authorize.conf with the calculation.

Any Help would be Appreciated!

Thanks in advance!

[diogofgm]

Check your limits.conf. Its recommended to change the limits on your ES SH for the scheduler:

[scheduler]
auto_summary_perc = 100
max_searches_perc = 75

Additionally, you can check the monitoring console for searches that are taking too long to run. Since there are several moving parts in ES (correlation search, data model acceleration searches, etc) slow searches can have an impact on your system. Also take look at you indexer layer since it can also contribute to search slowness.

If you post more details (infrastructure sizing, # correlation searches, accelerated data models, etc. ) maybe we can help more.

[satyaallaparthi]

sizing is 25 GB per day. I have 2 Indexers (Cluster) 16 core and 32 GB RAM, 2 SH (cluster) 16 core and 32 GB RAM, 1 SH standalone for ES 32 core and 64 GB RAM.

All are in windows servers 2016
13 data models are accelerated and 16 correlation searches are enabled..

and Just want to know any calculations to keep limits.conf? If yes, Please let me know the formula and calculation for my Environment.

Thanks,

[lakshman239]

Assuming you want to try and improve situation till you add additional Indexers, you could do the following:

1. check the time interval for all your 16 corr.searches. Are they/most of them running at same time, e.g. */5 * * * * ? Can you spread them so that some run at 0th hour, some at 15min past the hour etc... [ this will also to spread the searches and give sufficient time for completion]
2. Run the searches in the ES search bar and using job inspector, check time taken for completion. Can you optimise the searches? [e.g. use of stats, tstats etc..]
3. Look at all searches [ saved and scheduled - for first run, you can ignore datamodel acceleration searches] in both ES and non-ES SH and try to apply the above 2 steps (where possible).
4. Whats' your data retention for datamodels? do you need to retain 1m or 6months etc..? If you don't need longer retention, you can adjust them to reduce disk storage and better IOPS.

[diogofgm]

So in total you have more search heads than you have indexers? That might be a problem... Also you're missing CM/LM/Deployer in your list of instances
whatever you're doing in the in the other SHs will also have impact into the IDX cluster performance wise and must be accounted for. Im guessing you might have warnings the the non-ES SHs complaining about performance?
Ad-Hoc search will have priority over other kind of searches (Scheduled, DMA, etc) so if you have a small index tier with several SHs searching there its a possibility that you'll end up with deferred/skipped searches. Consider increasing the number of indexers in your cluster.

Check splunk's recommended architectures:
https://www.splunk.com/pdfs/technical-briefs/splunk-validated-architectures.pdf

More details about search priority from docs:
https://docs.splunk.com/Documentation/Splunk/7.3.1/Report/Configurethepriorityofscheduledreports#The...

For more details on the scheduler check this .conf presentation:https://conf.splunk.com/files/2017/slides/making-the-most-of-the-splunk-scheduler.pdf

[satyaallaparthi]

Yes, I have a CM/deployer/Shdeployer with 16 core and 32GB RAM.. I suggested my manger for another Indexer but they are agreeing.. any best solution ?