topic Re: Slow IO when Splunk enabled in Splunk Enterprise

Slow IO when Splunk enabled

techniclab — Tue, 11 Apr 2017 10:24:27 GMT

When Splunk is enabled (no searches are running) all io operations are slow. For example : vi takes 1 second to open.
Output of iostat

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.39    0.00    0.74    0.84    0.00   95.02

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             128.27       631.71      1215.79    1809915    3483345
dm-0             90.20       629.08      1214.34    1802375    3479205
dm-1              0.04         0.37         0.00       1068          0
dm-2              0.05         0.24         0.73        694       2093

Re: Slow IO when Splunk enabled

martin_mueller — Tue, 11 Apr 2017 10:43:40 GMT

based on that iostat output, your system and IO are doing fine - hardly any iowait going on, and lots of idle.

Re: Slow IO when Splunk enabled

techniclab — Tue, 11 Apr 2017 10:46:10 GMT

Yes and this is very strange, there is also timeouts when sshing into the host and changing users.

Re: Slow IO when Splunk enabled

martin_mueller — Tue, 11 Apr 2017 11:08:22 GMT

A common thing to check would be ulimits, by default they're set way too low for splunk - at least these: https://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements#Considerations_regarding_system-wide_resource_limits_on_.2Anix_systems

Re: Slow IO when Splunk enabled

techniclab — Tue, 11 Apr 2017 11:21:05 GMT

Thank you! I increased open files limit and splunk became blazing fast.

Re: Slow IO when Splunk enabled

jkat54 — Tue, 11 Apr 2017 11:24:20 GMT

Thats exactly what I was going to say... ULIMITS! Disabling THP gives you a good boost in performance as well!