Splunk Forwarder behavior

[smahtha]

Two questions:

1. Does Splunk forwarder maintain some kind of log files (or for that matter anything) which might keep growing in size and hog disk space.

2. How does Splunk forwarder reads a file. Does it keeps the file open or it periodicaly opens them and then closes them. We want to understand whether Splunk forwarder will be invisible to our own processes of deleting older files and wont disrupt existing processes by keeping open handles to the files?

[proctorgeorge · Accepted solution]

1. Yes Splunk forwarders have log files. They are located at \SplunkUniversalForwarder\var\log\splunk. The two most active ones are metrics.log and splunkd.log, metrics usually grows at a constant rate while splunkd will grow fast if there are errors that go unfixed. By default these files will grow to 25mb and then will be renamed to metrics.log.1, metrics.log.2, metrics.log.3, etc... by default this will go until 5 such files are present and then will start to delete the oldest one. Thus they can take up anywhere from 125mb to 150mb for each log type (though again, only metrics.log will get constant activity, on our average forwarder after 6 months of use all the other log files combined are less then 1mb). This log size and the amount of rollover files is adjustable in the $SPLUNK_HOME/etc/log.cfg configuration file so it is really up to you how much space they use.

2. Check out Dwaddle's response to this question for more insight but I am not confident enough in the internal workings to give a more detailed answer. But Splunk seems to do both depending on how fast the file is being modified.