They are not "steps". They're separate checks/measures to perform to try to alleviate delayed searches. See https://help.splunk.com/en/splunk-enterprise/get-started/deployment-capacity-manual/9.4/performance-reference/reference-hardware and https://help.splunk.com/en/splunk-enterprise-security-8/install/8.1/planning/minimum-specifications-for-a-production-deployment Enterprise Security is a very resource-intensive application. Therefore, it is recommended to install ES on a separate Splunk instance. It can, however, share indexers with other search heads. This should require no explanation. Real-time searches pin themselves to a CPU, preventing other searches from running there. Don't use real-time searches. See https://help.splunk.com/en/splunk-cloud-platform/search/search-manual/9.2.2406/search-and-report-in-real-time/about-real-time-searches-and-reports for more. Entire books could be written on making searches more efficient. Splunk has one at https://help.splunk.com/en/splunk-enterprise/search/search-manual/9.4/optimizing-searches/about-search-optimization Allow Skew gives the search scheduler permission to adjust the run time of a scheduled search to one with fewer other searches scheduled. Search Windows allow the scheduler to delay the start of a scheduled search in the event that resources are not yet available. See https://help.splunk.com/en/splunk-enterprise/create-dashboards-and-reports/reporting-manual/10.0/report-management/offset-scheduled-search-start-times and https://www.splunk.com/en_us/blog/platform/schedule-windows-vs-skewing.html There is a strong tendency among Splunk users to run their scheduled searches at the top of an hour. At most of the customers I've visited, this accounts for about half of all scheduled searches and is a source of most of their delayed and skipped searches. It also doesn't account for the 30-90 seconds of delay between when an event is generated and when it is searchable by Splunk. It's far better to use a cron schedule to have the search run at 2-3 minutes after the hour. Other peak search periods to avoid are 15, 30, and 45 minutes into any hour of the day. Splunk's Workload Management feature gives Splunk admins some control over how resource contention (CPU and memory) is handled. It also can be used to stop long-running searches, prevent real-time searches, and prevent users from running searches during peak times. See https://help.splunk.com/en/splunk-enterprise/administer/manage-workloads/9.4/workload-management-overview/about-workload-management for details. I've seen plenty of instances where a reports run once a day or even every week at 8 or 9 in the morning. This usually is unnecessary and takes away "slots" from other searches. Instead, these types of "batch" reports should run in less busy times of day such as 3am or on weekends.
... View more
