Adding a hot and cold storage solution to a distributed clustered splunk environment

[ASGrover]

Hi everyone,

I’m currently working with a Splunk distributed clustered environment (v9.4.1), with 3 indexers, 3 search heads and 1 cluster master, on RHEL. 

I recently added a second 500GB disk to each indexer in order to separate hot/warm and cold bucket storage. I have set up and mounted the 500GB disks hoping that should differentiate between the /indexes and the /coldstore. 

I also edited the indexes.conf file on the cluster master, an example is shown below:

[bmc]
homePath = /indexes/bmc/db
coldPath = /coldstore/bmc/colddb
thawedPath = $SPLUNK_DB/bmc/thaweddb
repFactor = auto
maxDataSize = auto_high_volume

I then applied the cluster-bundle as well as gave it a rolling-restart just in case. 

Even though (I think) that I have configured everything correctly, when I navigate to the cluster master GUI and go to the path 

Settings → Indexer Clustering → Indexes

The indexes tab is empty, with none of the default indexes or the custom indexes that I had made.

Has anyone encountered this behaviour where indexes do not appear in the Clustering UI, despite valid indexes.conf and bundle deployment?

[PrewinThomas]

@ASGrover 

Can you check bundle deployment status on the CM
splunk show cluster-bundle-status

Verify your indexes.conf is placed correctly
Eg:
$SPLUNK_HOME/etc/master-apps/<your_app>/local/indexes.conf

Verify index config is available in the indexer, run this in one of the indexer and verify
splunk btool indexes list bmc --debug

Does your new index have any data? If not, try with some test data
| makeresults | eval foo="bar" | collect index=bmc

Also did you find any errors on the CM _internal?

Lastly perform a restart on CM as well.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[livehybrid]

Hi @ASGrover 

Are you able to confirm that the indexers have been updated correctly on the indexers?

One way to check this is with btool:

$SPLUNK_HOME/bin/splunk cmd btool indexes list --debug

Also, are your peers (indexers) showing up in the Peers tab on the Indexer Clustering page from your cluster manager?

Lastly - Just double check you are on the cluster manager! I have found myself looking a other hosts before wondering where on earth my hosts have gone!

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[ASGrover]

When I use the btool command that you provided me with, what exactly do I look for? Because there is an overwhelming amount of information that is provided when I use that btool command. 

I can see my peers (indexers) in the Peers tab on the Indexer Clustering page from my cluster manager. 

And I have triple checked that I am on the cluster manager, I've often made the same mistake or looking at other hosts hahaha