Splunk Search

Realtime searches in ES delayed?

Splunk Employee
Splunk Employee

Hi,

I've installed Enterprise Security dedicated search head following all the best practices with beefy enough hardware specs and when running a simple realtime search, compared to a Splunk standard search head I'm getting a 60s+ delay before I get the same result.

What is happening?

1 Solution

Splunk Employee
Splunk Employee

ES app is using by default Indexed real-time_search

To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.

/opt/SPLUNK/6.0.3-ES-SH/splunk/bin $ ./splunk btool limits list realtime --debug
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   [realtime]
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/default/limits.conf alerting_period_ms = 3000
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   blocking = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   default_backfill = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   disabled = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   enforce_time_order = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_cluster_update_interval = 30
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_default_span = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_maximum_span = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexed_realtime_use_by_default = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexfilter = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             max_blocking_secs = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             queue_size = 10000

This is a necessary compromise between having the result in a matter of seconds vs 1+ minute and having lots of 'realtime' searches able to run concurrently.

To shorten the delay, you can attempt to decrease paramater

/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60

However this shouldn't be set to a value lower than 30 (seconds).

If in doubt, please contact Splunk Technical Support.

View solution in original post

Splunk Employee
Splunk Employee

ES app is using by default Indexed real-time_search

To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.

/opt/SPLUNK/6.0.3-ES-SH/splunk/bin $ ./splunk btool limits list realtime --debug
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   [realtime]
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/default/limits.conf alerting_period_ms = 3000
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   blocking = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   default_backfill = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   disabled = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   enforce_time_order = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_cluster_update_interval = 30
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_default_span = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_maximum_span = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexed_realtime_use_by_default = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexfilter = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             max_blocking_secs = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             queue_size = 10000

This is a necessary compromise between having the result in a matter of seconds vs 1+ minute and having lots of 'realtime' searches able to run concurrently.

To shorten the delay, you can attempt to decrease paramater

/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60

However this shouldn't be set to a value lower than 30 (seconds).

If in doubt, please contact Splunk Technical Support.

View solution in original post

Contributor

One of my customers is building its ES environment, and have real trouble with that realtime delay.
What is the meaning of reducing "indexed_realtime_disk_sync_delay" to the minimum (for example 1)

Why should it be at least 30?

Thanks.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!