Deployment Architecture

Change location of introspection index

BrendanCO
Path Finder

Hello. I'd like to change the location of this disk hogging index. I've read through some other posts on this and it refers to an indexes.conf that doesn't reside where they say it does. Here are the ones I have:

find . -name "indexes.conf"

./opt/splunk/etc/master-apps/_cluster/default/indexes.conf
./opt/splunk/etc/system/default/indexes.conf
./opt/splunk/etc/system/local/indexes.conf
./opt/splunk/etc/apps/sample_app/default/indexes.conf
./opt/splunk/etc/apps/SplunkLightForwarder/default/indexes.conf

The one that has the type of information I'm looking for (location to where it write) is ./opt/splunk/etc/apps/sample_app/default/indexes.conf, contents are:

Version 6.5.3

Creates a sample index for sample data.

[sample]
homePath = $SPLUNK_DB/sample/db
coldPath = $SPLUNK_DB/sample/colddb
thawedPath = $SPLUNK_DB/sample/thaweddb

That doesn't seem like the path I'm looking for. Can anyone help point me in the right direction, please? I feel like this should be configurable in the GUI but can't find anything there on that.

Thanks in advance.

Tags (1)
0 Karma

adonio
Ultra Champion

not sure why would you like to change the location of that index but in the case you need to, you can edit the path as you posted in your question. create a new inputs.conf in /opt/splunk/etc/system/local (this is highest precedence in splunk file structure)
in that file, indicate where you would like the introspection index to be:
[_introspection]
homePath = path/to/index/_introspection/db
coldPath = path/to/index/_introspection/colddb
thawedPath = path/to/index/_introspection/thaweddb
more to read here: https://docs.splunk.com/Documentation/Splunk/6.5.3/Admin/Indexesconf
you can find where $SPLUNK_DB is pointing to by navigating to settings -> server settings -> General Settings -> scroll down to "path to indexes field"

0 Karma

BrendanCO
Path Finder

Thanks Adonio! The reasoning behind my wanting to change the location is simply disk space on the primary filesystem is growing day by day. It's now at 72% use and grows by an entire percent per day. What's odd is that I put an ln -s for the dispatch folder to go to the new filesystem and my utilization hasn't changed in days on the target filesystem. Only on the primary.

Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 5.5G 2.2G 72% /
devtmpfs 3.9G 68K 3.9G 1% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/xvdb1 40G 12G 27G 30% /splunkdata

So the /dev/xvda1 filesystem is where /opt/splunk resides. The /dev/xvdb1 filesystem is where things are supposed to go but has remained at 30% use for this entire week. So, something isn't working right! This is what prompted me to want to move indexes, unless you advise against that!
I need to get my hands around this before I run out of space on /dev/xvda1. Once I get that set up correctly, then I can start looking at how to manage log retention and not be such a bother on this board... 🙂

0 Karma

adonio
Ultra Champion

look at the last part of my answer and at your indexes configurations, if they contain $SPLUNK_DB the index location reffers to twhere $SPLUNK__DB points. you can change $SPLUNK_DB
if you would like to move all indexes to new file system, follow this link: https://docs.splunk.com/Documentation/Splunk/6.5.3/Indexer/Moveanindex

BrendanCO
Path Finder

Using your instructions, I was able to move all my indexes to my new filesystem. It is reflecting the new location in the GUI as well. Thanks again, Adonio. As always, a great help.

0 Karma

adonio
Ultra Champion

is it a clustered environment or single indexer?

0 Karma

BrendanCO
Path Finder

Single indexer

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...