is it possible to change the log rotation timing for the internal logs that Universal Forwarder and Heavy Forwarder output to the OS.
For example, splunkd.log. Currently, the logs are rotated by file size, but can we rotate on a daily basis.
is it possible ?
The appender parameters are size-based. You'd have to try and see if the DailyRollingFileAppender is available instead and reconfigure whole logging framework for the other appender.
Thank you for your quick response. could you please share some insights regarding "DailyRollingFileAppender" ?
To be fully honest, I'm just guessing since the configuration looks like log4j but as far as I know splunk itself doesn't use java so there must be simply some similar framework. (and there are many very similar framework for other programming languages/environments)
With log4j you have whole framework where on the one side you define logging categories and all the stuff responsible for ingesting the logged events and on the other side you define output channels (kinda like in splunk itself ;-)) - appenders. They append entries to some "storage channel" - files, network sockets for sending over to external server, console and so on.
As I can see in the log.cfg in default splunk installation, it uses an appender called RollingFileAppender. This one as you can see here https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html (again - it's a log4j doc whereas splunk uses something different but similar) the options here only govern file size and maximum number of backlog files. But if you configure a DailyRollingFileAppender - https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html - you can configure other parameters pertaining to time-based rotation.
But.
1) I don't know if this appender is implemented in the library splunk uses
2) time-based rotation makes it possible for splunk to fill up the disk in case of a very "talkative" splunk operation (like some bug or something).