topic Re: Change location of introspection index in Deployment Architecture

Change location of introspection index

BrendanCO — Tue, 29 Sep 2020 13:49:25 GMT

Hello. I'd like to change the location of this disk hogging index. I've read through some other posts on this and it refers to an indexes.conf that doesn't reside where they say it does. Here are the ones I have:

find . -name "indexes.conf"

./opt/splunk/etc/master-apps/_cluster/default/indexes.conf
./opt/splunk/etc/system/default/indexes.conf
./opt/splunk/etc/system/local/indexes.conf
./opt/splunk/etc/apps/sample_app/default/indexes.conf
./opt/splunk/etc/apps/SplunkLightForwarder/default/indexes.conf

The one that has the type of information I'm looking for (location to where it write) is ./opt/splunk/etc/apps/sample_app/default/indexes.conf, contents are:

Version 6.5.3

Creates a sample index for sample data.

[sample]
homePath = $SPLUNK_DB/sample/db
coldPath = $SPLUNK_DB/sample/colddb
thawedPath = $SPLUNK_DB/sample/thaweddb

That doesn't seem like the path I'm looking for. Can anyone help point me in the right direction, please? I feel like this should be configurable in the GUI but can't find anything there on that.

Thanks in advance.

Re: Change location of introspection index

adonio — Tue, 25 Apr 2017 13:54:31 GMT

is it a clustered environment or single indexer?

Re: Change location of introspection index

BrendanCO — Tue, 25 Apr 2017 15:06:00 GMT

Single indexer

Re: Change location of introspection index

adonio — Tue, 29 Sep 2020 13:49:47 GMT

not sure why would you like to change the location of that index but in the case you need to, you can edit the path as you posted in your question. create a new inputs.conf in /opt/splunk/etc/system/local (this is highest precedence in splunk file structure)
in that file, indicate where you would like the introspection index to be:
[_introspection]
homePath = path/to/index/_introspection/db
coldPath = path/to/index/_introspection/colddb
thawedPath = path/to/index/_introspection/thaweddb
more to read here: https://docs.splunk.com/Documentation/Splunk/6.5.3/Admin/Indexesconf
you can find where $SPLUNK_DB is pointing to by navigating to settings -> server settings -> General Settings -> scroll down to "path to indexes field"

Re: Change location of introspection index

BrendanCO — Tue, 25 Apr 2017 17:32:01 GMT

Thanks Adonio! The reasoning behind my wanting to change the location is simply disk space on the primary filesystem is growing day by day. It's now at 72% use and grows by an entire percent per day. What's odd is that I put an ln -s for the dispatch folder to go to the new filesystem and my utilization hasn't changed in days on the target filesystem. Only on the primary.

Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 5.5G 2.2G 72% /
devtmpfs 3.9G 68K 3.9G 1% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/xvdb1 40G 12G 27G 30% /splunkdata

So the /dev/xvda1 filesystem is where /opt/splunk resides. The /dev/xvdb1 filesystem is where things are supposed to go but has remained at 30% use for this entire week. So, something isn't working right! This is what prompted me to want to move indexes, unless you advise against that!
I need to get my hands around this before I run out of space on /dev/xvda1. Once I get that set up correctly, then I can start looking at how to manage log retention and not be such a bother on this board... 🙂

Re: Change location of introspection index

adonio — Tue, 29 Sep 2020 13:49:58 GMT

look at the last part of my answer and at your indexes configurations, if they contain $SPLUNK_DB the index location reffers to twhere $SPLUNK__DB points. you can change $SPLUNK_DB
if you would like to move all indexes to new file system, follow this link: https://docs.splunk.com/Documentation/Splunk/6.5.3/Indexer/Moveanindex

Re: Change location of introspection index

BrendanCO — Tue, 25 Apr 2017 18:49:28 GMT

Using your instructions, I was able to move all my indexes to my new filesystem. It is reflecting the new location in the GUI as well. Thanks again, Adonio. As always, a great help.