How do i set frozenTimePeriodInSec to specefic index

[splunkgk]

Hi,
I wanted to apply a retention policy on a specific index which where i wanted to set frozenTimePeriodInSec = 31540000 and maxsize to 50 G. I have set these attributed in my index setting under $SPLUNK_HOME/etc/apps/local/indexes.conf.

After restarting the splunk service, i am getting below message as

/opt/splunk/etc/apps/launcher/local/indexes.conf, line 16: frozenTimePeriodInSec  (value:  31540000).
                Invalid key in stanza [ops_metrics] in /opt/splunk/etc/apps/launcher/local/indexes.conf, line 51: frozenTimePeriodInSec  (value:  31540000).
                Invalid key in stanza [email] in /opt/splunk/etc/system/local/alert_actions.conf, line 4: reportServerURL  (value:  ).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunk/splunk-6.5.2-67571ef4b87d-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

What i am doing wrong here? Can some one help to set the required parameters on selected indexes?

my index settings are here below

[bod_access]
coldPath = $SPLUNK_DB/bod_access/colddb
homePath = $SPLUNK_DB/bod_access/db
thawedPath = $SPLUNK_DB/bod_access/thaweddb
bucketRebuildMemoryHint = 0
compressRawdata = 1
enableDataIntegrityControl = 0
enableOnlineBucketRepair = 1
enableTsidxReduction = 1
minHotIdleSecsBeforeForceRoll = 0
rtRouterQueueSize =
rtRouterThreads =
suspendHotRollByDeleteQuery = 0
syncMeta = 1
timePeriodInSecBeforeTsidxReduction = 345600
frozenTimePeriodInSec = 31540000
maxTotalDataSizeMB = 50000

thanks

[maciep]

Not sure if it's this simple, but that setting is plural: frozenTimePeriodInSecs

[splunkgk]

Hi Maciep, Yes you are correct. Its my mistake. The setting must be in plural. Thank you for catching this.
Meanwhile, I have a question on altering indexes.conf file.

1. for Specific indexes altering (like setting roll bucket, max size) is the $SPLUNK_HOME/etc/apps//local/indexes.conf is the file where is need to edit? and

2. If i need to set the bucket roll policy and setting maxSize data on all of my indexes, is $SPLUNK_HOME/etc/system/local/inidexes.conf is the place where is need settings?

-Thanks

[maciep]

Yes, I believe you would make those changes in indexes.conf. If you want to apply settings to all of your indexes, then you can put those settings in the default stanza. If you need to alter any of those default settings, you can put them in the stanza for the individual index - these settings will win over the default settings.

But like esix mentions, where you put your indexes.conf file(s) is really up to you. Yes, it needs to be on your indexers. And yes, how you get it there depends on whether you have an indexer cluster or just standalone indexers.

But generally speaking, for any conf files, it's important to understand the order of precedence. You can have the same conf file in multiple places with conflicting settings - but if you know the precedence, you should be good.

So yes, you can put the setting etc/system/local. Or you can create your own app like esix mentioned and put your indexes.conf in there. We, for example, have one app where all of our indexes are defined.

[esix_splunk]

The answer here depends on a few things..

1) Are you in a clustered environment?
2) Do you have an app naming strategy in place?

If you're not in a clustered environment, all settings for indexes could be placed in $splunk_home$/etc/system/local/indexes.conf. Otherwise, they need to be placed on the cluster Master node, under $splunk_home$/etc/master-apps/.

Now, #2.. this relates to configurations in both the clustered and non-clustered environment. In large deployments, it's recommneded (best) practiced to name apps and apply configurations based on their purpose in the organization.

So you can have an app... called index-bod_access. In this, you'd have an indexes.conf with all your settings. This would look like :

$splunk_home$/etc/apps/index-bod_access/local/indexes.conf

[bod_access]
coldPath = xxxxxxx
homePath = xxxxxx
thawedPath = xxxxx
etc

This of course would go on an indexer, in a non-clustered environment. You would apply this app in the same manner in a cluster by applying this to the cluster Master's $splunk_home$/etc/master-apps folder, and applying the cluster bundle.

[splunkgk]

Hey Esix, Thanks for reply-
No, I do not have a clustered indexes environment, But i have a app naming strategy in place.
All my apps are listed under $SPLUNK_HOME/etc/apps/. The individual indexes.conf are placed under each apps like $SPLUNK_HOME/etc/apps/app-name/local/indexes.conf.

So, As i mentioned in my question, i have app name called "bod_accesses" and i have set the below settings in $splunk_home$/etc/apps/index-bod_access/local/indexes.conf
[bod_access]
coldPath = $SPLUNK_DB/bod_access/colddb
homePath = $SPLUNK_DB/bod_access/db
thawedPath = $SPLUNK_DB/bod_access/thaweddb
bucketRebuildMemoryHint = 0
compressRawdata = 1
enableDataIntegrityControl = 0
enableOnlineBucketRepair = 1
enableTsidxReduction = 1
minHotIdleSecsBeforeForceRoll = 0
rtRouterQueueSize =
rtRouterThreads =
suspendHotRollByDeleteQuery = 0
syncMeta = 1
timePeriodInSecBeforeTsidxReduction = 345600
frozenTimePeriodInSecs = 31540000
maxTotalDataSizeMB = 50000

Now, after i restated the splunk services, How do i make sure i have only 1 year events in HOT DB and rest of events in COLD DB.
Secondly , The HOT DB folder become empty after restart. Before there was Latest event are 3 years and now this is blank. Where does data went? is that deleted since i did not mentioned frozed db path in settings?
I need here to set as HOT/WARM DB as 1 year and rest should go to cold DB then after 3 months this must roll to FREEZ bucket.

-Thanks

[esix_splunk]

So first, when you restart SPlunk, your data is rolled from HOT, to Warm. Remember here, HOT buckets are writable. Warm, not writable. And your HOT/WARM share the same volume.

So I think for you, your best friend will be the 'dbinspect' command and the Fire Brigade App on splunkbase.

dbinspect index=bod_access

Look at the fields: state and startEpoch and endEpoch, path.. You can get information out of there easily as your volumes and event times are in the buckets..

[splunkgk]

Hi Esix,
I am not getting the results for index which i applied FrozenTimePeriodInSecs = 31540000 with dbinspect command. its resulting with blank.

And also observed that, the COLDDB became empty after rolling this. How do i make sure after rolling from HOT, the data must be in COLD for 3 months?

-Thanks

[woodcock]

Yes, that is it.