topic metrics.log filling up filesystem on LWF in Deployment Architecture

metrics.log filling up filesystem on LWF

Branden — Tue, 21 Sep 2010 21:57:15 GMT

We have a LWF on Linux that is forwarding to our indexer. We're a little tight on space, but in my experience the LWFs don't use up too much space.

On this particular LWF, the metrics.log file is growing at a much higher rate than our AIX-based LWFs. Now we're at risk of filling up the file system. Is there a easy/safe way to clean up these metrics.log files? Is there a reason that file would be growing much faster on Linux than on AIX?

Thanks!

Re: metrics.log filling up filesystem on LWF

southeringtonp — Tue, 21 Sep 2010 22:18:31 GMT

The logs should automatically rotate and delete copies older than metrics.log.5. You can tune the file size for rotation and the number of copies retained in $SPLUNK_HOME/etc/log.cfg

By default, metrics.log should be rotating after 25 MB, and keep 5 backup copies, so it would cap at 150 MB.

Splunk indexes those files, so it should be safe to delete all but the most recent metrics.log. You can always go back to the _internal index to get the data. (At least for standalone -- I think this is still true for the LWF).

Re: metrics.log filling up filesystem on LWF

Branden — Tue, 21 Sep 2010 22:24:02 GMT

Thank you very much for the info!

Re: metrics.log filling up filesystem on LWF

gkanapathy — Wed, 22 Sep 2010 11:12:56 GMT

Every 30 seconds, metrics outputs a line for each of:

the top ten (by default) sources
the top ten (by default) sourcetypes

These items are the ones that are most likely to be of variable number across different lightweight forwarders. There are many others, but e.g., there is usually only one host processed on a lightweight forwarder, the same number of queues and pipelines on each, no network inputs on any of them, the same number of outbound connections, etc.

If one forwarder has only a single source or sourcetype, while the other has ten or more, the latter's log file will grow noticeably more quickly. The other factors could affect this, but it would be much less common for them to be different.