Indexer _internal size under /opt/splunk/var/lib is large

[jordanking1992]

Hello,

We have an indexer cluster that has a custom indexes.conf that specifies the volume path and retention of each index.
However, it appears the _internal DB on each of the indexers it writing to /opt/splunk/var/lib instead of our custom volume where all the other indexes are writing. This is causing our /opt/splunk filesystem to fill up.

Can someone explain why the indexers are not sending their _internaldb logs to /opt/splunk_hot even though we are referencing the volume in _internal? The ellipses are all the other indexes in the indexes.conf

indexes.conf

# VOLUME SETTINGS
# One Volume for Hot and Cold
[volume:primary]
path = /opt/splunk_hot

[volume:secondary]
path = /opt/splunk_cold
....
[_internal]
repFactor = auto
homePath   = volume:primary/_internal/db
coldPath   = volume:secondary/_internaldb/colddb
thawedPath = /opt/splunk_cold/_internaldb/thaweddb

[codebuilder]

Assuming the config you provided is what you've actually deployed, the answer is hiding in plain sight 🙂

There is an error in your homePath declaration.

 homePath   = volume:primary/_internal/db

Should be:

 homePath   = volume:primary/_internaldb/db

[jordanking1992]

Hey, thanks for the reply. However, Splunk doesn't care what the name of the directory is as long as that directory never changes. When an index is created using [], Splunk knows to write and read data from the directory paths specified under this stanza.

The solution to my problem was what @sathwikr076 mentioned. By changed the DB_PATH under /opt/splunk/etc/splunk-launch.conf, data for the _internal index began writing to splunk_hot like anticipated.

Thank you for your response though!

[codebuilder]

Ah, good to know! And glad you got it resolved!

[FrankVl]

I guess there is another config file somewhere that takes precedence. Try btool to see what the effective configuration is for that index and what file it comes from.

[sathwikr076 · Accepted solution]

Hello,

did you mention your primary volume path at /opt/splunk/etc/splunk-launch.conf. if you did not mention your path their, the buckets will go to the default location which is /opt/splunk/var/lib/splunk.
Thanks.

[jordanking1992]

Hey, i had no idea about this setting. So if i put the SPLUNK_DB path to /opt/splunk_hot, will that affect any of my other indexes? Keep in mind, all indexes excluding _internal are correctly sending to /opt/splunk_hot. I would've assume explicitly setting the path's above (which are not referencing SPLUNK_DB) woulve work...

[sathwikr076]

Can you let me know if there is any difference in indexes.conf between _internal index and other indexes which is why they are going to your actual path.
yes, if you add Splunk_DB path it will apply to all the indexes.