Hi,
I've installed Enterprise Security dedicated search head following all the best practices with beefy enough hardware specs and when running a simple realtime search, compared to a Splunk standard search head I'm getting a 60s+ delay before I get the same result.
What is happening?
ES app is using by default Indexed real-time_search
To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.
/opt/SPLUNK/6.0.3-ES-SH/splunk/bin $ ./splunk btool limits list realtime --debug
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf [realtime]
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/default/limits.conf alerting_period_ms = 3000
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf blocking = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf default_backfill = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf disabled = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf enforce_time_order = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_cluster_update_interval = 30
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_default_span = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_disk_sync_delay = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_maximum_span = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf indexed_realtime_use_by_default = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf indexfilter = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf max_blocking_secs = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf queue_size = 10000
This is a necessary compromise between having the result in a matter of seconds vs 1+ minute and having lots of 'realtime' searches able to run concurrently.
To shorten the delay, you can attempt to decrease paramater
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_disk_sync_delay = 60
However this shouldn't be set to a value lower than 30 (seconds).
If in doubt, please contact Splunk Technical Support.
ES app is using by default Indexed real-time_search
To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.
/opt/SPLUNK/6.0.3-ES-SH/splunk/bin $ ./splunk btool limits list realtime --debug
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf [realtime]
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/default/limits.conf alerting_period_ms = 3000
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf blocking = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf default_backfill = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf disabled = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf enforce_time_order = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_cluster_update_interval = 30
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_default_span = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_disk_sync_delay = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_maximum_span = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf indexed_realtime_use_by_default = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf indexfilter = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf max_blocking_secs = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf queue_size = 10000
This is a necessary compromise between having the result in a matter of seconds vs 1+ minute and having lots of 'realtime' searches able to run concurrently.
To shorten the delay, you can attempt to decrease paramater
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf indexed_realtime_disk_sync_delay = 60
However this shouldn't be set to a value lower than 30 (seconds).
If in doubt, please contact Splunk Technical Support.
One of my customers is building its ES environment, and have real trouble with that realtime delay.
What is the meaning of reducing "indexed_realtime_disk_sync_delay" to the minimum (for example 1)
Why should it be at least 30?
Thanks.