Splunk Search

Realtime searches in ES delayed?

abonuccelli_spl
Splunk Employee
Splunk Employee

Hi,

I've installed Enterprise Security dedicated search head following all the best practices with beefy enough hardware specs and when running a simple realtime search, compared to a Splunk standard search head I'm getting a 60s+ delay before I get the same result.

What is happening?

1 Solution

abonuccelli_spl
Splunk Employee
Splunk Employee

ES app is using by default Indexed real-time_search

To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.

/opt/SPLUNK/6.0.3-ES-SH/splunk/bin $ ./splunk btool limits list realtime --debug
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   [realtime]
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/default/limits.conf alerting_period_ms = 3000
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   blocking = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   default_backfill = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   disabled = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   enforce_time_order = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_cluster_update_interval = 30
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_default_span = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_maximum_span = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexed_realtime_use_by_default = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexfilter = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             max_blocking_secs = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             queue_size = 10000

This is a necessary compromise between having the result in a matter of seconds vs 1+ minute and having lots of 'realtime' searches able to run concurrently.

To shorten the delay, you can attempt to decrease paramater

/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60

However this shouldn't be set to a value lower than 30 (seconds).

If in doubt, please contact Splunk Technical Support.

View solution in original post

abonuccelli_spl
Splunk Employee
Splunk Employee

ES app is using by default Indexed real-time_search

To lessen the impact on the indexer, you can enable indexed real-time search. This will basically run the search like a historical search, but will also continually update it with new events as they appear on disk.

/opt/SPLUNK/6.0.3-ES-SH/splunk/bin $ ./splunk btool limits list realtime --debug
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   [realtime]
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/default/limits.conf alerting_period_ms = 3000
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   blocking = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   default_backfill = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   disabled = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   enforce_time_order = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_cluster_update_interval = 30
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_default_span = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_maximum_span = 0
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexed_realtime_use_by_default = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/limits.conf   indexfilter = 1
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             max_blocking_secs = 60
/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             queue_size = 10000

This is a necessary compromise between having the result in a matter of seconds vs 1+ minute and having lots of 'realtime' searches able to run concurrently.

To shorten the delay, you can attempt to decrease paramater

/opt/SPLUNK/6.0.3-ES-SH/splunk/etc/system/default/limits.conf                             indexed_realtime_disk_sync_delay = 60

However this shouldn't be set to a value lower than 30 (seconds).

If in doubt, please contact Splunk Technical Support.

ehudb
Contributor

One of my customers is building its ES environment, and have real trouble with that realtime delay.
What is the meaning of reducing "indexed_realtime_disk_sync_delay" to the minimum (for example 1)

Why should it be at least 30?

Thanks.

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...