Getting Data In

retention policy on log files

marone
Explorer

Hi, I want to implement retention policy on log files, in the doc https://docs.splunk.com/Documentation/Splunk/8.0.3/Troubleshooting/Enabledebuglogging they didn't mention such a configuration, there is only how to configure the maximum size of a log file (configuration of log-local.cfg), i want this configuration be applied to log files who live in SPLUNK_HOME/var/log, is there any workaround to do so ?

Labels (1)
0 Karma
1 Solution

venkateshparank
Path Finder

You can add below frozenTimePeriodInSecs in your index stanza.
Example: If you want to retain the logs for 1 day then frozenTimePeriodInSecs = 86400

[_internal]

frozenTimePeriodInSecs = 86400

View solution in original post

PavelP
Motivator

Hello @marone,

which log files do you have in $SPLUNK_HOME/etc ? Most of logs are under $SPLUNK_HOME/var/log

0 Karma

marone
Explorer

thanks for your comment, yes a typo error logs are in $SPLUNK_HOME/var/log and the log-local.cfg are in $SPLUNK_HOME/etc i modified it

0 Karma

PavelP
Motivator

@marone, I think you asking about log file rotation and autodeletion, right? This is configured under log.cfg:

appender.rootAppender=ConsoleAppender
appender.rootAppender.layout=PatternLayout
appender.rootAppender.layout.ConversionPattern=%d{%m-%d-%Y %H:%M:%S.%l %z} %-5p %c - %m%n
# if these log files are getting too big for your liking, turn down the maxFileSize.
# it's best to not make them too small, however, because these logs can be very
# useful in troubleshooting.
appender.A1=RollingFileAppender
appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log
appender.A1.maxFileSize=25000000 # default: 25MB (specified in bytes).
appender.A1.maxBackupIndex=5
appender.A1.layout=PatternLayout

as you can see, for most log files that is max 5 log files x 25MB. More for license log.

Is this the info that you looked for?

0 Karma

marone
Explorer

yes i saw the doc was looking for a parameter like frozenTimePeriodInSecs in indexes.conf, but i think decreasing maxFilesize and number of backup index plus configuring frozenTimePeriodInSecs for _internalwill result to a good retention policy i guess

0 Karma

venkateshparank
Path Finder

You can add below frozenTimePeriodInSecs in your index stanza.
Example: If you want to retain the logs for 1 day then frozenTimePeriodInSecs = 86400

[_internal]

frozenTimePeriodInSecs = 86400

marone
Explorer

thanks for reply, i's true what you are saying but your answer is applied to bucket stored in $SPLUNK_HOME/var/lib/splunk, but the information wil still in log files who lives in SPLUNK_HOME/var/log as I said in my question, i want to apply retention policy on those logs (configure log-local.cfg if it's possible or another way), or applying retention policy as you said for index will automaitcally delete data from logs (aka from SPLUNK_HOME/var/log) ?

0 Karma

venkateshparank
Path Finder

All logs under SPLUNK_HOME/var/log will be ingested into _internal index.
Events in this index are kept for 6 years by default.
If you want to change the retention period, you can modify the same for frozenTimePeriodInSecs in indexes.conf for _internal index. Hope this helps and please accept if this helps.

Below is the example:

[_internal]
homePath = $SPLUNK_DB/_internaldb/db
coldPath = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
maxDataSize = 100
frozenTimePeriodInSecs = 2419200

0 Karma

marone
Explorer

yes indeed, i was looking for a parameter in log-config.cfg like frozenTimePeriodInSecs but i guess there is not, instead of that configure frozenTimePeriodInSecs in indexes.conf plus reduce amount of file size for all log files will do the job for a good retention policy. Thanks for mentioning that most of logs files are ingested to _internal index

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @marone,
you can set a retention policy for each index (default is 6 years).
you can do this settinf the parameter frozenTimePeriodInSecs in indexes.conf.

You can find more infos at https://docs.splunk.com/Documentation/Splunk/8.0.4/Indexer/Setaretirementandarchivingpolicy

Ciao.
Giuseppe

marone
Explorer

thanks for reply, i's true what you are saying but your answer is applied to bucket stored in $SPLUNK_HOME/var/lib/splunk, but the information wil still in log files who lives in SPLUNK_HOME/var/log as I said in my question, i want to apply retention policy on those logs (configure log-local.cfg if it's possible or another way), or applying retention policy as you said for index will automaitcally delete data from logs (aka from SPLUNK_HOME/var/log) ?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @marone,
let me understand: you're speaking of Splunk log files (e.g. splunkd.log.1 2 3 4 5) is it correct?

After they are written on disk, they are read by Splunk and stored in _internal index, this means that you can delete the splunkd.log.1 2 3 4 5 files.

If instead you want to manage retention on _internal index, you have to change the frozenTimePeriodInSecs parameter in $SPLUNK_HOME/system/local/indexes.conf, if not present create it copying the _internal stanza in default folder.

Ciao.
Giuseppe

0 Karma

marone
Explorer

thanks i see what you mean

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...