metrics.log filling up filesystem on LWF

[Branden]

We have a LWF on Linux that is forwarding to our indexer. We're a little tight on space, but in my experience the LWFs don't use up too much space.

On this particular LWF, the metrics.log file is growing at a much higher rate than our AIX-based LWFs. Now we're at risk of filling up the file system. Is there a easy/safe way to clean up these metrics.log files? Is there a reason that file would be growing much faster on Linux than on AIX?

Thanks!

[gkanapathy]

Every 30 seconds, metrics outputs a line for each of:

• the top ten (by default) sources
• the top ten (by default) sourcetypes

These items are the ones that are most likely to be of variable number across different lightweight forwarders. There are many others, but e.g., there is usually only one host processed on a lightweight forwarder, the same number of queues and pipelines on each, no network inputs on any of them, the same number of outbound connections, etc.

If one forwarder has only a single source or sourcetype, while the other has ten or more, the latter's log file will grow noticeably more quickly. The other factors could affect this, but it would be much less common for them to be different.

[southeringtonp · Accepted solution]

The logs should automatically rotate and delete copies older than metrics.log.5. You can tune the file size for rotation and the number of copies retained in $SPLUNK_HOME/etc/log.cfg

By default, metrics.log should be rotating after 25 MB, and keep 5 backup copies, so it would cap at 150 MB.

Splunk indexes those files, so it should be safe to delete all but the most recent metrics.log. You can always go back to the _internal index to get the data. (At least for standalone -- I think this is still true for the LWF).

[Branden]

Thank you very much for the info!