topic Re: Index path not updating correctly in Deployment Architecture

Index path not updating correctly

Kozanic — Wed, 10 Jan 2018 23:28:40 GMT

Splunk Version 6.6.2
I am getting lack of space errors due to poor set-up of our Splunk environment and am trying to resolve, but having issues.

The error I'm currently receiving (there were others, but this seems to be the last one) is below:

Search peer server3 has the following message: Disk Monitor: The index processor has paused data flow. Current free disk space on partition '/' has fallen to 4492MB, below the minimum of 5000MB. Data writes to index path '/opt/splunkhot/_audit/db'cannot safely proceed. Increase free disk space on partition '/' by removing or relocating data.

Steps taken so far:

I have followed the steps outlined here "https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Moveanindex" to move the indexes to a new location.
I have also updated /opt/splunk/etc/splunk-launch.conf with the following: SPLUNK_DB=/var/lib/db/splunkhot
Splunk has been restarted on indexes and cluster master yet I'm still seeing the above error after restarting.

Not sure what else to check / update.

Update:

So.... I have determined the cause of my issue - but now I'm not sure of the best steps to resolve.

Within master-apps, I have a indexes app which is defining a number of apps specifically - eg not using $SPLUNK_DB in the path.

How does one update this without breaking index integrity? Under normal processes, one would shutdown the indexer, relocate the indexes and update the path, restart service and all good.

But when I deploy the app to update - this will restart the services on the index servers automatically - not giving me a chance to copy the indexes.
Can I copy prior to pushing the update? or is there a method of deploying where the services are not restarted automatically?

Re: Index path not updating correctly

HiroshiSatoh — Thu, 11 Jan 2018 00:25:23 GMT

If you change the following settings, the error disappears. Restart is necessary.
But is it correct that the capacity of the disk is 5 G or less?

settings » Server settings » General settings
Pause indexing if free disk space (in MB) falls below *

Re: Index path not updating correctly

Kozanic — Thu, 11 Jan 2018 00:30:02 GMT

HI HiroshiSatoh,

I did see this setting, unfortunately I cannot update it as the current space being used is that shared with the OS and total disk space is down to 5Gb free - hence the attempts to move the indexes.

Re: Index path not updating correctly

HiroshiSatoh — Thu, 11 Jan 2018 01:24:05 GMT

Looking at the message, SPLUNK_DB is not "/var/lib/db/splunkhot", but is the setting OK?
Have you restarted?

Data writes to index path '/opt/splunkhot/_audit/db'cannot safely proceed.

Re: Index path not updating correctly

micahkemp — Thu, 11 Jan 2018 02:01:17 GMT

You may try clearing the messages to see if they return. I've seen instances where the messages will not clear on their own.

Re: Index path not updating correctly

mayurr98 — Thu, 11 Jan 2018 05:52:43 GMT

Hi @Kozanic

well, you have installed Splunk on the /opt and you're using the /opt as well for your indexes. Read about the indexes.conf how to change the homePath for your indexes.

To move your existing data to another location proceed like this:

1) stop Splunk

2) change the homePath in indexes.conf

3)move all existing data from old homePath (in your case /opt/splunkhot/_audit/db) to new homePath

4)start Splunk

In a cluster environment you should do this on cluster master and then push indexes.conf to all indexers.
I hope this helps .

After moving the data you may have to clean eventdata :

1)Stop Splunk. Erase all your logs under $SPLUNK_HOME/var/log/splunk

2)Open Command Line and cd to the $SPLUNK_HOME/bin directory.

3)Type ./splunk clean eventdata

4)Enter your splunk admin / password

5)Start Splunk

I hope this helps!

Re: Index path not updating correctly

Kozanic — Tue, 29 Sep 2020 17:38:01 GMT

Thanks for the help mayurr98.

I have checked indexes.conf - homePath on all indexes is set to $SPLUNK_DB/IndexName/db/.

As mentioned, I have updated $SPLUNK_DB via splunk-launch.conf on both indexes in my cluster - yet I'm still seeing some indexes that are using the old path.

When running | dbinspect index=yourindex I get the below. seems to indicate that internal indexes are holding the old path where as others are picking up the new path.
_telemetry - /opt/splunkhot/_telemetry/db/db_1515589234
_introspection - /opt/splunkhot/_introspection/db/rb_1515619431
dlm_uberagent_log - /var/lib/db/splunkhot/dlm_uberAgent_log/db/rb_1515619434
dlm_uberagent - /var/lib/db/splunkhot/dlm_uberagent/db/rb_1515575095

Not sure why the internals are holding the old path.

Is there another spot I need to check for config??

Re: Index path not updating correctly

Kozanic — Thu, 11 Jan 2018 23:13:06 GMT

Messages return after clearing.

I also see them when I run the health check from Monitoring Console

Re: Index path not updating correctly

mayurr98 — Fri, 12 Jan 2018 03:53:38 GMT

Are you forwarding the internal events of the master to indexers? (recommended). When you did that, only have to have care about the location in the indexers.

You have to configure in the indexes.conf a new location for _audit _internal _introspection and the best way is using volume definition.

Do this in a the master-apps/_cluster/local/indexes.conf

Certainly copy the default stanza to local/indexes.conf and change the home path

Re: Index path not updating correctly

Kozanic — Mon, 15 Jan 2018 00:51:11 GMT

A few times

Re: Index path not updating correctly

micahkemp — Mon, 15 Jan 2018 01:31:14 GMT

You need to either:

A: define master-apps/app/default/indexes.conf in an app named appropriately so that it takes precedence over these other apps. Maybe prefix them with A_ or 1_

B: define master-apps/app/local/indexes.conf, which will take precedence over any default/indexes.conf.

Consult the Configuration file precedence documentation to see the full explanation of which app/config takes precedence.

In my opinion it's wrong for 3rd party apps to configure index paths without making use of the environment variable.

Re: Index path not updating correctly

Kozanic — Tue, 16 Jan 2018 04:44:11 GMT

After Speaking with Splunk support in conjunction with the details outlined here: "https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Moveanindex" steps taken were as follows:

Set Cluster Master into maintenance mode
spot index server 1, apply required updates to index app and splunk-launch.conf to point indexes to correct location
copy index folders from old location to new location
Start index server- confirm start correctly
repeat steps 2-4 on remaining index servers
remove cluster master from maintenance mode
Update index app within master apps
re-deploy updated app

After following these steps I can confirm that all indexes now pointing to the correct spots with no issues.