topic Re: retention policy on log files in Getting Data In

retention policy on log files

marone — Sun, 07 Jun 2020 01:29:22 GMT

Hi, I want to implement retention policy on log files, in the doc https://docs.splunk.com/Documentation/Splunk/8.0.3/Troubleshooting/Enabledebuglogging they didn't mention such a configuration, there is only how to configure the maximum size of a log file (configuration of log-local.cfg), i want this configuration be applied to log files who live in SPLUNK_HOME/var/log, is there any workaround to do so ?

Re: retention policy on log files

gcusello — Tue, 26 May 2020 08:58:48 GMT

Hi @marone,
you can set a retention policy for each index (default is 6 years).
you can do this settinf the parameter frozenTimePeriodInSecs in indexes.conf.

You can find more infos at https://docs.splunk.com/Documentation/Splunk/8.0.4/Indexer/Setaretirementandarchivingpolicy

Ciao.
Giuseppe

Re: retention policy on log files

venkateshparank — Tue, 26 May 2020 09:00:58 GMT

You can add below frozenTimePeriodInSecs in your index stanza.
Example: If you want to retain the logs for 1 day then frozenTimePeriodInSecs = 86400

[_internal]

frozenTimePeriodInSecs = 86400

Re: retention policy on log files

PavelP — Tue, 26 May 2020 09:02:01 GMT

Hello @marone,

which log files do you have in $SPLUNK_HOME/etc ? Most of logs are under $SPLUNK_HOME/var/log

Re: retention policy on log files

marone — Tue, 26 May 2020 09:18:06 GMT

thanks for your comment, yes a typo error logs are in $SPLUNK_HOME/var/log and the log-local.cfg are in $SPLUNK_HOME/etc i modified it

Re: retention policy on log files

marone — Tue, 26 May 2020 09:28:52 GMT

thanks for reply, i's true what you are saying but your answer is applied to bucket stored in $SPLUNK_HOME/var/lib/splunk, but the information wil still in log files who lives in SPLUNK_HOME/var/log as I said in my question, i want to apply retention policy on those logs (configure log-local.cfg if it's possible or another way), or applying retention policy as you said for index will automaitcally delete data from logs (aka from SPLUNK_HOME/var/log) ?

Re: retention policy on log files

marone — Tue, 26 May 2020 09:29:01 GMT

Re: retention policy on log files

venkateshparank — Wed, 30 Sep 2020 05:31:55 GMT

All logs under SPLUNK_HOME/var/log will be ingested into _internal index.
Events in this index are kept for 6 years by default.
If you want to change the retention period, you can modify the same for frozenTimePeriodInSecs in indexes.conf for _internal index. Hope this helps and please accept if this helps.

Below is the example:

[_internal]
homePath = $SPLUNK_DB/_internaldb/db
coldPath = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
maxDataSize = 100
frozenTimePeriodInSecs = 2419200

Re: retention policy on log files

gcusello — Wed, 30 Sep 2020 05:31:58 GMT

Hi @marone,
let me understand: you're speaking of Splunk log files (e.g. splunkd.log.1 2 3 4 5) is it correct?

After they are written on disk, they are read by Splunk and stored in _internal index, this means that you can delete the splunkd.log.1 2 3 4 5 files.

If instead you want to manage retention on _internal index, you have to change the frozenTimePeriodInSecs parameter in $SPLUNK_HOME/system/local/indexes.conf, if not present create it copying the _internal stanza in default folder.

Ciao.
Giuseppe

Re: retention policy on log files

PavelP — Tue, 26 May 2020 09:54:12 GMT

@marone, I think you asking about log file rotation and autodeletion, right? This is configured under log.cfg:

appender.rootAppender=ConsoleAppender
appender.rootAppender.layout=PatternLayout
appender.rootAppender.layout.ConversionPattern=%d{%m-%d-%Y %H:%M:%S.%l %z} %-5p %c - %m%n
# if these log files are getting too big for your liking, turn down the maxFileSize.
# it's best to not make them too small, however, because these logs can be very
# useful in troubleshooting.
appender.A1=RollingFileAppender
appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log
appender.A1.maxFileSize=25000000 # default: 25MB (specified in bytes).
appender.A1.maxBackupIndex=5
appender.A1.layout=PatternLayout

as you can see, for most log files that is max 5 log files x 25MB. More for license log.

Is this the info that you looked for?

Re: retention policy on log files

marone — Tue, 26 May 2020 10:29:41 GMT

yes i saw the doc was looking for a parameter like frozenTimePeriodInSecs in indexes.conf, but i think decreasing maxFilesize and number of backup index plus configuring frozenTimePeriodInSecs for _internalwill result to a good retention policy i guess

Re: retention policy on log files

marone — Tue, 26 May 2020 10:39:57 GMT

yes indeed, i was looking for a parameter in log-config.cfg like frozenTimePeriodInSecs but i guess there is not, instead of that configure frozenTimePeriodInSecs in indexes.conf plus reduce amount of file size for all log files will do the job for a good retention policy. Thanks for mentioning that most of logs files are ingested to _internal index

Re: retention policy on log files

marone — Tue, 26 May 2020 12:17:10 GMT

thanks i see what you mean