It looks like splunk is causing a open
syscall once per second for those
files.
The periodic opens on dispatch directories, especially in the absence of any search load, are likely caused by the reaper process that checks for jobs with expired TTLs.
By default, the reaper triggers every 30 seconds. You can make it trigger less often by adding this to your limits.conf:
[search]
# Reap every 10 minutes instead of every 30 seconds.
dispatch_dir_cleanup_freq = 600
I suppose it is possible that the reaper is generating so much load that it is starving search processes of I/O to the NFS mount, though that wouldn't be my first guess for a root cause here.
I tried changeing the
poll.interval.rebuild and
poll.interval.check settings in the
[pooling] section of server.conf. But
that did not seem to have an impact.
I wouldn't expect these settings to have much effect on search performance, as search in a separate process must always read all conf files anew.
poll.interval.check controls how often Splunk checks conf files for updates. It doesn't affect how often Splunk touches dispatch directories.
poll.interval.rebuild does not affect disk I/O at all. It controls how often Splunk rebuilds in-memory conf datastructures.
... View more
