When Splunk is enabled (no searches are running) all io operations are slow. For example : vi takes 1 second to open.
Output of iostat
avg-cpu: %user %nice %system %iowait %steal %idle
3.39 0.00 0.74 0.84 0.00 95.02
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
vda 128.27 631.71 1215.79 1809915 3483345
dm-0 90.20 629.08 1214.34 1802375 3479205
dm-1 0.04 0.37 0.00 1068 0
dm-2 0.05 0.24 0.73 694 2093
A common thing to check would be ulimits, by default they're set way too low for splunk - at least these: https://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements#Considerations_re...
A common thing to check would be ulimits, by default they're set way too low for splunk - at least these: https://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements#Considerations_re...
Thank you! I increased open files limit and splunk became blazing fast.
Thats exactly what I was going to say... ULIMITS! Disabling THP gives you a good boost in performance as well!
based on that iostat output, your system and IO are doing fine - hardly any iowait going on, and lots of idle.
Yes and this is very strange, there is also timeouts when sshing into the host and changing users.