Splunk Enterprise

Slow IO when Splunk enabled

techniclab
Engager

When Splunk is enabled (no searches are running) all io operations are slow. For example : vi takes 1 second to open.
Output of iostat

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.39    0.00    0.74    0.84    0.00   95.02

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             128.27       631.71      1215.79    1809915    3483345
dm-0             90.20       629.08      1214.34    1802375    3479205
dm-1              0.04         0.37         0.00       1068          0
dm-2              0.05         0.24         0.73        694       2093
Tags (1)
0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

A common thing to check would be ulimits, by default they're set way too low for splunk - at least these: https://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements#Considerations_re...

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

A common thing to check would be ulimits, by default they're set way too low for splunk - at least these: https://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements#Considerations_re...

techniclab
Engager

Thank you! I increased open files limit and splunk became blazing fast.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Thats exactly what I was going to say... ULIMITS! Disabling THP gives you a good boost in performance as well!

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

based on that iostat output, your system and IO are doing fine - hardly any iowait going on, and lots of idle.

0 Karma

techniclab
Engager

Yes and this is very strange, there is also timeouts when sshing into the host and changing users.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...