How to move internal indexes from the default location to a new directory in an indexer clustering environment?

[Esky73]

I have a distributed search environment consisting of 2 x indexers, 1 x master node, and a search head.

Currently, I am running out of disk space due to the _internal indexes which are on a smallish partition in the default location $SPLUNK_HOME/var/lib/splunk

I have a directory /app/splunk/data which has lots of free space and where all my app indexes are - I understand these are in this location because the indexes take precedence in the /etc/slave-apps/app/local/ directories.

I'm uncertain which index I need to change to move the _internal dbs from the default to the /app/splunk/data directory?

My understanding from the documentation is that best practice would be to make this change in the $SPLUNK_HOME/etc/master-apps/_cluster/local on the master node and push the config out?

[jmallorquin · Accepted solution]

Hi,

Are you forwarding the internal events of the master to indexers? (recommended). When you did that, only have to have care about the location in the indexers.

You have to configure in the indexes.conf a new location for _audit _internal _introspection and the best way is using volume definition.

And correct, is in the master-apps/_cluster/local/indexes.conf

Hope i help you

[Esky73]

Doing a bit more research i found this article which says i am on the right track.

https://answers.splunk.com/answers/85446/how-to-move-internaldb-to-a-new-partition-in-a-clustered-in...

However - ive realised that if i make the change on the master node and its pushed out to all indexers - if the new location for the internal indexers is /app/splunk/data and that partition only exists on my 2 x indexers and not on the master node itself - will that cause an issue ?

Would adding indexes.conf to each indexer (only) pointing to the new location work ?