Slow IO when Splunk enabled

[techniclab]

When Splunk is enabled (no searches are running) all io operations are slow. For example : vi takes 1 second to open.
Output of iostat

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.39    0.00    0.74    0.84    0.00   95.02

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             128.27       631.71      1215.79    1809915    3483345
dm-0             90.20       629.08      1214.34    1802375    3479205
dm-1              0.04         0.37         0.00       1068          0
dm-2              0.05         0.24         0.73        694       2093

[martin_mueller · Accepted solution]

A common thing to check would be ulimits, by default they're set way too low for splunk - at least these: https://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements#Considerations_re...

[techniclab]

Thank you! I increased open files limit and splunk became blazing fast.

[jkat54]

Thats exactly what I was going to say... ULIMITS! Disabling THP gives you a good boost in performance as well!

[martin_mueller]

based on that iostat output, your system and IO are doing fine - hardly any iowait going on, and lots of idle.

[techniclab]

Yes and this is very strange, there is also timeouts when sshing into the host and changing users.