_audit and _internal index data retention

[ualbanytech]

EDIT: Splunk version = 4.1.6

Are there any guidelines on the length of time that _audit and _internal index data should be kept?

I have come up with age-out policies for our Splunk events, however
the part I'm stuck on is how long should I keep my _audit and _internal events?

My initial thought is to keep events in those two indexes for the same age as my oldest index (5 years).

The only problem is the majority of my indexes are only retained for 1 year or less.

Spacewise, it seems wasteful to keep all of _audit and _internal for 5 years.

[splunkreal]

Solved this with maxTotalDataSizeMB

[SamHTexas]

Thank u for your post. Am asked for a document to prove that Splunk Audit logs are kept for 1 year. Where do I find such a document & edit it if necessary? Thank u in advance.

[x3mboy]

I had changed the $SPLUNK_HOME/etc/system/local/indexes.conf to change _internal and _audit size, but when I try the bundle-push, it fails saying "No new bundle will be pushed. The master and peers already have this bundle with bundle id = xxxxx"

How changes in $SPLUNK_HOME/etc/system/local/ should be pushed to the indexers?

[rtev]

Make the changes in $SPLUNK_HOME/etc/master-apps/local/indexes.conf on the Master and Splunk should recognize it needs a new bundle.

[splunksriniwipr]

Hi,
splunk maintains its default settings in $SPLUNK_HOME/etc/system/default path...

If you want to make any changes on default properties, then you can create inputs.conf or index.conf etc conf files under /etc/system/local/ direcotry....

use same stanza's in *.conf files. with different values... Hope It will helpful

Thanks,
Srinivas

[buckiboy]

Guys, are we sure its called _audit. Looking at our indexers, the directory is called audit.

[tomasmoser]

Yes, it's correct. See below.

splunk@test:/opt/splunk/var/lib/splunk$ splunk btool indexes list _audit | grep audit
[_audit]
coldPath = $SPLUNK_DB/audit/colddb
homePath = $SPLUNK_DB/audit/db
thawedPath = $SPLUNK_DB/audit/thaweddb
tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary

I agree it's not logical and Splunk should change directory name from "audit" to "_audit" on a filesystem.

[hexx · Accepted solution]

You certainly don't need to keep events from _internal and _audit for 6 years.

Events in _internal mostly are indexed from $SPLUNK_HOME/var/log/splunk. The majority of the volume comes from files such as splunkd.log and metrics.log.

The information contained in those events is typically interesting to troubleshoot Splunk-specific issues or to get sample measurements of event-processing thruput from metrics.log.

As it is rare to have to troubleshoot Splunk issues that are older than a month, I would say that the default retention period of 28 days set for _internal in $SPLUNK_HOME/etc/system/default/indexes.conf is adequate :

[_internal]
homePath = $SPLUNK_DB/_internaldb/db
coldPath = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
maxDataSize = 100
frozenTimePeriodInSecs = 2419200

The _audit index is where Splunk logs events from fschange inputs by default (see the File system change monitor section of inputs.conf.spec for more information - http://www.splunk.com/base/Documentation/latest/Admin/Inputsconf).

Events in this index are kept for 6 years by default (again, a setting inherited from $SPLUNK_HOME/etc/system/default/indexes.conf), but unless you have your own fschange inputs, only $SPLUNK_HOME/etc is audited in this way. For that reason, you could want to shorten the retention period for this index, although it is usually very small in size anyway.

Example : Let's modify $SPLUNK_HOME/etc/system/local/indexes.conf to set a retention period of 20 days for _internal and 60 days for _audit. We'll simply add the two following stanzas to that file :

[_internal]
frozenTimePeriodInSecs = 1728000

[_audit]
frozenTimePeriodInSecs = 5184000

[ualbanytech]

I wasn't clear enough. I did not change the defaults for _internal yet the default
frozenTimePeriodInSecs has been exceeded by over a year.

It seems my problem relates to how Splunk ages out data (only when rolling between buckets).
And, that is contingent on other settings.

The age out based on time is too complicated.

Based on your answer, I decided to just set the max size to 1 GB and 2 GB for _audit and _internal (respectively).

Thank You very much!

[hexx]

You still need to declare the stanza for which you are changing the parameters from the default. I have amended my answer above to provide a clear example of what should go into the local version of indexes.conf.

[ualbanytech]

Follow up question. I created my local policies by placing indexes.conf in

SPLUNK_HOME/etc/system/local

I just verified the default/indexes.conf has the policies as you outlined for _internal and _audit

However, my _internal and _audit indexes do not appear to be obeying the policies.

I did not re-create stanzas in my local indexes.conf as it was my understanding that
any I define in local overrides those in default dir.

Manager >> Indexes shows
_audit 3,548 MB w/ earliest Dec 30, 2009
_internal 5,193 MB w/ earliest Dec 4, 2009