Getting Data In

Indexer _internal size under /opt/splunk/var/lib is large

jordanking1992
Path Finder

Hello,

We have an indexer cluster that has a custom indexes.conf that specifies the volume path and retention of each index.
However, it appears the _internal DB on each of the indexers it writing to /opt/splunk/var/lib instead of our custom volume where all the other indexes are writing. This is causing our /opt/splunk filesystem to fill up.

Can someone explain why the indexers are not sending their _internaldb logs to /opt/splunk_hot even though we are referencing the volume in _internal? The ellipses are all the other indexes in the indexes.conf

indexes.conf

# VOLUME SETTINGS
# One Volume for Hot and Cold
[volume:primary]
path = /opt/splunk_hot

[volume:secondary]
path = /opt/splunk_cold
....
[_internal]
repFactor = auto
homePath   = volume:primary/_internal/db
coldPath   = volume:secondary/_internaldb/colddb
thawedPath = /opt/splunk_cold/_internaldb/thaweddb
0 Karma
1 Solution

sathwikr076
Communicator

Hello,

did you mention your primary volume path at /opt/splunk/etc/splunk-launch.conf. if you did not mention your path their, the buckets will go to the default location which is /opt/splunk/var/lib/splunk.
Thanks.

View solution in original post

codebuilder
Influencer

Assuming the config you provided is what you've actually deployed, the answer is hiding in plain sight 🙂

There is an error in your homePath declaration.

 homePath   = volume:primary/_internal/db

Should be:

 homePath   = volume:primary/_internaldb/db
----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

jordanking1992
Path Finder

Hey, thanks for the reply. However, Splunk doesn't care what the name of the directory is as long as that directory never changes. When an index is created using [], Splunk knows to write and read data from the directory paths specified under this stanza.

The solution to my problem was what @sathwikr076 mentioned. By changed the DB_PATH under /opt/splunk/etc/splunk-launch.conf, data for the _internal index began writing to splunk_hot like anticipated.

Thank you for your response though!

0 Karma

codebuilder
Influencer

Ah, good to know! And glad you got it resolved!

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

FrankVl
Ultra Champion

I guess there is another config file somewhere that takes precedence. Try btool to see what the effective configuration is for that index and what file it comes from.

0 Karma

sathwikr076
Communicator

Hello,

did you mention your primary volume path at /opt/splunk/etc/splunk-launch.conf. if you did not mention your path their, the buckets will go to the default location which is /opt/splunk/var/lib/splunk.
Thanks.

jordanking1992
Path Finder

Hey, i had no idea about this setting. So if i put the SPLUNK_DB path to /opt/splunk_hot, will that affect any of my other indexes? Keep in mind, all indexes excluding _internal are correctly sending to /opt/splunk_hot. I would've assume explicitly setting the path's above (which are not referencing SPLUNK_DB) woulve work...

0 Karma

sathwikr076
Communicator

Can you let me know if there is any difference in indexes.conf between _internal index and other indexes which is why they are going to your actual path.
yes, if you add Splunk_DB path it will apply to all the indexes.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...