Splunk Enterprise

Sanity check on Indexer cluster with differing peer-apps/indexes.conf & system/local/indexes.conf?

bamflpn
New Member

Sanity check please re: Splunk indexer remote storage configuration

(1) Our indexers are set up in a clustered environment, and originally frozen buckets were moved to a local external location using coldToFrozenPath.

(2) We are now trying to use a script to have all frozen buckets written to AWS, and  the app containing the updated indexes.conf was distributed using the master node

  • Script to move / freeze buckets to AWS is working, validated again by checking date / time stamps of last written files

(3) However, we are STILL getting "duplicate" buckets being written to the local storage instance, even though everything SHOULD only be written to AWS

  • When looking at the pertinent indexes in the web GUI, the Frozen Path is still set, even though it has been removed completely from the deployed app's indexes.conf, and verified on the indexers in /peer-apps/xxxx/local/indexes.conf
  • Trying to manually remove the frozen path in the GUI doesn't work, "not allowed in a clustered environment"

(4) When trying to find out the 5W's, I've noticed that there is an /opt/splunk/etc/system/local/indexes.conf that ALSO exists on each indexer, and this version still uses the Frozen Path to the local storage.

Question:

Since the deployed peer-apps indexes.conf with the script does seem to be working correctly, is it safe (relative) for me to remove the /system/local/indexes.conf?

Labels (1)
0 Karma

danspav
SplunkTrust
SplunkTrust

Hi @bamflpn,

The peer-apps/xxx/local directory has higher precedence than etc/system/local on indexers.

(see docs: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles#Precedenc...)

To confirm the location of the setting that's overwriting your coldToFrozenPath, you can run btool on your indexers:

./splunk btool indexes list --debug | grep coldToFrozenPath

 

That should print out each line of all indexes.conf on the host that contribute to the final indexes config for the coldToFrozen option.

Find the line that has the wrong value and that's the file that you should update. 

 

If you want to delete the system/local/indexes.conf file you should first check what it's doing.

To see what other config is being used from system/local/indexes.conf you can run:

./splunk btool indexes list --debug | grep system/local

Any config that is listed will be applied to that indexer from the system/local/indexes.conf file. Either move the config to a peer-apps app, or just remove it if you don't need it.

 

Cheers,
Daniel

0 Karma

bamflpn18
Explorer

Sir:

Perfect, thank you very much.  

Comparing the grep'ed results for system/local coldToFrozenDir vs peer-apps coldToFrozenScript showed me the ones I missed. 

0 Karma
Get Updates on the Splunk Community!

Unlock New Opportunities with Splunk Education: Explore Our Latest Courses!

At Splunk Education, we’re dedicated to providing top-tier learning experiences that cater to every skill ...

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...