topic Re: High IOPS on target NAS in Monitoring Splunk

High IOPS on target NAS

selvig — Thu, 06 Dec 2012 19:17:48 GMT

We're running Splunk 4.3.3 on a Linux box. The target log files are on a NetApp NAS, accessed by Splunk through an NFS mount. The target log files are Java application server log4j logs. The naming convention is such that the current log is always -.log, and Splunk is set to use that as its data source, eg it is only targetting that one file per data input, as opposed to looking at everything after the /.
What we're seeing is that the Splunk user account shows up as the top IOPS consumer on the NetApp. Why is this so high, and are there any ways to reduce this? We could move to another OS for the Splunk indexers, as there's some thought that Solaris might read the logs more efficiently. Is there any advantage in using followtail = 0 versus followtail = 1? Any other suggestions?

Re: High IOPS on target NAS

RicoSuave — Thu, 06 Dec 2012 20:31:14 GMT

Follow tail and when to use it has been discussed at great lengths here: http://splunk-base.splunk.com/answers/57819/when-is-it-appropriate-to-set-followtail-to-true

As for the IOPS well this will be normal since you are reading data from the NAS. This should be mostly Read operations and if this log file is big and this is the first time indexing it then splunk will read it as fast as it can. If this is a forwarder you can limit how fast it reads the file, subsequently lessening the load by adjusting the thruput attribute in limits.conf. On a forwarder it defaults to 256, unlimited on indexer.

Re: High IOPS on target NAS

RicoSuave — Thu, 06 Dec 2012 20:47:37 GMT

I answered first, so accept this answer. 😉