When increasing retention time on index in master-apps/_cluster/local/indexes.conf, why did the changes not apply to slaves?

[duke_splunk_adm]

I increased the retention time of an index from 30 days to 13 months on the cluster master, in
$SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf
and applied the bundle.

Splunk shows cluster-bundle-status then showed that the bundle had been pushed, and the new retention size shows up on the cluster in $SPLUNK_HOME/etc/slave-apps/_cluster/local/indexes.conf but this change did NOT get propagated to my regular indexes.conf, which we keep in $SPLUNK_HOME/etc/apps/oit-indexes/local/indexes.conf for historical reasons.

The apply-bundle command on the master did not restart the indexers, either. I tried simply offline->restart, but the information was not copied.

When I create a completely new index and run the same commands, the index shows up there as expected, but this retention expansion did not.

Do I need to edit the local indexes.conf by hand for this change? I'm not completely averse to doing so, I just want to make sure that I'm not missing something.

[somesoni2]

I don't think the changes that you did in master-apps/cluster will go and update your indexes.conf kept in other app. For that you need to update that app in master-apps/oid-indexes... path. Please note that Within the slave-apps/[local|default] directories, the special _cluster subdirectory has a higher precedence than any app subdirectories starting with a lowercase letter (for example, anApp). However, it has a lower precedence than any apps starting with an uppercase letter (for example, AnApp). This is due to the location of the underscore ("") character in the lexicographical order. So, your changes should be applied properly (check in UI if you can or from search head using | rest /services/data/indexes search) as your custom app starts with lower case and is overridden with _cluster copy of indexes.conf.

[ddrillic]

@duke_splunk_admins, you can also run ./splunk btool indexes list --debug on the indexer and see the logical indexes and their sources. Something like -

$ ./splunk btool indexes list --debug | more
/opt/splunk/etc/slave-apps/_cluster/local/indexes.conf   [_audit]
/opt/splunk/etc/system/default/indexes.conf              assureUTF8 = false
/opt/splunk/etc/system/default/indexes.conf              bucketRebuildMemoryHint = auto
/opt/splunk/etc/system/default/indexes.conf              coldPath = $SPLUNK_DB/audit/colddb
/opt/splunk/etc/system/default/indexes.conf              coldPath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf              coldToFrozenDir = 
/opt/splunk/etc/system/default/indexes.conf              coldToFrozenScript = 
/opt/splunk/etc/system/default/indexes.conf              compressRawdata = true
/opt/splunk/etc/system/default/indexes.conf              defaultDatabase = main
/opt/splunk/etc/system/default/indexes.conf              enableDataIntegrityControl = false
/opt/splunk/etc/system/default/indexes.conf              enableOnlineBucketRepair = true
/opt/splunk/etc/system/default/indexes.conf              enableRealtimeSearch = true
/opt/splunk/etc/slave-apps/_cluster/local/indexes.conf   enableTsidxReduction = true

[duke_splunk_adm]

Perfect, that is exactly what I needed.

[ddrillic]

Great - you can easily see the index set-up and from where the configuration parameters came from - most useful ; -)

[adonio]

hello duke_splunk_admins,
when pushing from /etc/master-apps/ the apps (folders) lands in /etc/slave-apps/ on the Slave Indexer.
look for your configuration in $SPLUNK_HOME/etc/slave-apps/_cluster/local/indexes.conf
read more about Indexer clustering here:
http://docs.splunk.com/Documentation/Splunk/6.5.3/Indexer/Updatepeerconfigurations
also slave-apps takes precedence in file hierarchy so configurations should apply regardless. here is the order:
1. Slave-apps local directory
2. System local directory
3. Apps local directory
4. Slave apps default
5. apps default
6. system default

hope it helps

[duke_splunk_adm]

I appreciate the hierarchy, thanks.

I do find it odd that adding a new index changes the app local indexes.conf but changing the retention doesn't. Maybe it will update the next time I add a new index.