Getting Data In

Indexer _internal size under /opt/splunk/var/lib is large

jordanking1992
Path Finder

Hello,

We have an indexer cluster that has a custom indexes.conf that specifies the volume path and retention of each index.
However, it appears the _internal DB on each of the indexers it writing to /opt/splunk/var/lib instead of our custom volume where all the other indexes are writing. This is causing our /opt/splunk filesystem to fill up.

Can someone explain why the indexers are not sending their _internaldb logs to /opt/splunk_hot even though we are referencing the volume in _internal? The ellipses are all the other indexes in the indexes.conf

indexes.conf

# VOLUME SETTINGS
# One Volume for Hot and Cold
[volume:primary]
path = /opt/splunk_hot

[volume:secondary]
path = /opt/splunk_cold
....
[_internal]
repFactor = auto
homePath   = volume:primary/_internal/db
coldPath   = volume:secondary/_internaldb/colddb
thawedPath = /opt/splunk_cold/_internaldb/thaweddb
0 Karma
1 Solution

sathwikr076
Communicator

Hello,

did you mention your primary volume path at /opt/splunk/etc/splunk-launch.conf. if you did not mention your path their, the buckets will go to the default location which is /opt/splunk/var/lib/splunk.
Thanks.

View solution in original post

codebuilder
SplunkTrust
SplunkTrust

Assuming the config you provided is what you've actually deployed, the answer is hiding in plain sight 🙂

There is an error in your homePath declaration.

 homePath   = volume:primary/_internal/db

Should be:

 homePath   = volume:primary/_internaldb/db
----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

jordanking1992
Path Finder

Hey, thanks for the reply. However, Splunk doesn't care what the name of the directory is as long as that directory never changes. When an index is created using [], Splunk knows to write and read data from the directory paths specified under this stanza.

The solution to my problem was what @sathwikr076 mentioned. By changed the DB_PATH under /opt/splunk/etc/splunk-launch.conf, data for the _internal index began writing to splunk_hot like anticipated.

Thank you for your response though!

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Ah, good to know! And glad you got it resolved!

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

FrankVl
Ultra Champion

I guess there is another config file somewhere that takes precedence. Try btool to see what the effective configuration is for that index and what file it comes from.

0 Karma

sathwikr076
Communicator

Hello,

did you mention your primary volume path at /opt/splunk/etc/splunk-launch.conf. if you did not mention your path their, the buckets will go to the default location which is /opt/splunk/var/lib/splunk.
Thanks.

View solution in original post

jordanking1992
Path Finder

Hey, i had no idea about this setting. So if i put the SPLUNK_DB path to /opt/splunk_hot, will that affect any of my other indexes? Keep in mind, all indexes excluding _internal are correctly sending to /opt/splunk_hot. I would've assume explicitly setting the path's above (which are not referencing SPLUNK_DB) woulve work...

0 Karma

sathwikr076
Communicator

Can you let me know if there is any difference in indexes.conf between _internal index and other indexes which is why they are going to your actual path.
yes, if you add Splunk_DB path it will apply to all the indexes.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!