Splunk Enteprise Delayed searches

[fedayn05]

Hello Team,

I wanna ask something that I really cannot figure out by myself , I have a splunk entreprise Installed on an  ubuntu with over 2 To , 32 Go of RAM and 38 CPU. 

With all these I still get so many delayed searches (up to 97%) : 

""The percentage of non high priority searches delayed (99%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance.""

I really do not know the reason , is it because we are using ES Security or what exactly.

Thank you for your time.

Kind regards,

[isoutamo]

Please, tell more about your architecture, deployment, splunk version, daily ingestion amount also how many searches (scheduled vs ad-hoc) you have.

[fedayn05]

Hello,

Thank you for your time.  I am using a single ubuntu 22.04 instance for Splunk  ubuntu with over 2 To , 32 Go of RAM and 38 CPU. 

I am using the 10.0.0 version , about the daily ingestion it is around 40 Go.

As i am new to splunk i did not get what you meant by searches (scheduled vs ad-hoc) 

[isoutamo]

I missed ES part earlier. It probably explains your issues. It depends on how many correlation searches and CIM accelerations are running your single box, but I suspect that your environment is too much for running it in single node.

You could find monitoring console under settings. There are links/icon for it. Just click it and then enable it from its setting tab/link. After that there are Search link where you could look those several dashboards which shows e.g. what are those deferred and skipped searches.

[gcusello]

Hi @fedayn05 ,

in addition to the checks hinted by @richgalloway ,

check the IOPS of your storage (e.g. using Bonnie++ or anothe tool) : probably this is the bottleneck.

Ciao.

Giuseppe

[fedayn05]

Hello,

Thank you for your time. I have checked this first and it was actually not the source of the problem.

Thank you for your time.

[richgalloway]

Searches are delayed because too many searches are trying to run at the same time.  There are a few things you can do about it.

1. Ensure all instances meet or exceed Splunk's Reference Hardware specification.
2. Ensure Splunk Enterprise Security runs on a dedicated Search Head (or SHC).  Do not run searches unrelated to ES on that SH.
3. Eliminate unneeded scheduled searches.
4. Prohibit real-time searches.
5. Verify all scheduled searches complete as quickly as possible.  Do this by minimizing their search windows and using efficient searches.
6. Implement Allow Skew and Schedule Windows.
7. Distribute search run times evenly across the clock.  Avoid running at peak times such as 0, 15, 30, or 45 minutes of each hour.
8. Consider using Workload Management to control the search behavior of users.
9. Move daily/weekly/monthly scheduled searches to off hours.

[fedayn05]

Hello,

Thank you for your time.

As I am a bit new to splunk , I would appreciate it if you can please explain a bit further these steps.

Thank you

[richgalloway]

They are not "steps".  They're separate checks/measures to perform to try to alleviate delayed searches.

1. See https://help.splunk.com/en/splunk-enterprise/get-started/deployment-capacity-manual/9.4/performance-... and https://help.splunk.com/en/splunk-enterprise-security-8/install/8.1/planning/minimum-specifications-...
2. Enterprise Security is a very resource-intensive application.  Therefore, it is recommended to install ES on a separate Splunk instance.  It can, however, share indexers with other search heads.
3. This should require no explanation.
4. Real-time searches pin themselves to a CPU, preventing other searches from running there.  Don't use real-time searches.  See https://help.splunk.com/en/splunk-cloud-platform/search/search-manual/9.2.2406/search-and-report-in-... for more.
5. Entire books could be written on making searches more efficient.  Splunk has one at https://help.splunk.com/en/splunk-enterprise/search/search-manual/9.4/optimizing-searches/about-sear...
6. Allow Skew gives the search scheduler permission to adjust the run time of a scheduled search to one with fewer other searches scheduled.  Search Windows allow the scheduler to delay the start of a scheduled search in the event that resources are not yet available.  See https://help.splunk.com/en/splunk-enterprise/create-dashboards-and-reports/reporting-manual/10.0/rep... and https://www.splunk.com/en_us/blog/platform/schedule-windows-vs-skewing.html
7. There is a strong tendency among Splunk users to run their scheduled searches at the top of an hour.  At most of the customers I've visited, this accounts for about half of all scheduled searches and is a source of most of their delayed and skipped searches.  It also doesn't account for the 30-90 seconds of delay between when an event is generated and when it is searchable by Splunk.  It's far better to use a cron schedule to have the search run at 2-3 minutes after the hour.  Other peak search periods to avoid are 15, 30, and 45 minutes into any hour of the day.
8. Splunk's Workload Management feature gives Splunk admins some control over how resource contention (CPU and memory) is handled.  It also can be used to stop long-running searches, prevent real-time searches, and prevent users from running searches during peak times.  See https://help.splunk.com/en/splunk-enterprise/administer/manage-workloads/9.4/workload-management-ove...for details.
9. I've seen plenty of instances where a reports run once a day or even every week at 8 or 9 in the morning.  This usually is unnecessary and takes away "slots" from other searches.  Instead, these types of "batch" reports should run in less busy times of day such as 3am or on weekends.