Index path not updating correctly

[Kozanic]

Splunk Version 6.6.2
I am getting lack of space errors due to poor set-up of our Splunk environment and am trying to resolve, but having issues.

The error I'm currently receiving (there were others, but this seems to be the last one) is below:

Search peer server3 has the following message: Disk Monitor: The index processor has paused data flow. Current free disk space on partition '/' has fallen to 4492MB, below the minimum of 5000MB. Data writes to index path '/opt/splunkhot/_audit/db'cannot safely proceed. Increase free disk space on partition '/' by removing or relocating data.

Steps taken so far:

• I have followed the steps outlined here "https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Moveanindex" to move the indexes to a new location.
• I have also updated /opt/splunk/etc/splunk-launch.conf with the following: SPLUNK_DB=/var/lib/db/splunkhot
• Splunk has been restarted on indexes and cluster master yet I'm still seeing the above error after restarting.

Not sure what else to check / update.

Update:

So.... I have determined the cause of my issue - but now I'm not sure of the best steps to resolve.

Within master-apps, I have a indexes app which is defining a number of apps specifically - eg not using $SPLUNK_DB in the path.

How does one update this without breaking index integrity? Under normal processes, one would shutdown the indexer, relocate the indexes and update the path, restart service and all good.

But when I deploy the app to update - this will restart the services on the index servers automatically - not giving me a chance to copy the indexes.
Can I copy prior to pushing the update? or is there a method of deploying where the services are not restarted automatically?

[Kozanic · Accepted solution]

After Speaking with Splunk support in conjunction with the details outlined here: "https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Moveanindex" steps taken were as follows:

1. Set Cluster Master into maintenance mode
2. spot index server 1, apply required updates to index app and splunk-launch.conf to point indexes to correct location
3. copy index folders from old location to new location
4. Start index server- confirm start correctly
5. repeat steps 2-4 on remaining index servers
6. remove cluster master from maintenance mode
7. Update index app within master apps
8. re-deploy updated app

After following these steps I can confirm that all indexes now pointing to the correct spots with no issues.

[micahkemp]

You need to either:

A: define master-apps/app/default/indexes.conf in an app named appropriately so that it takes precedence over these other apps. Maybe prefix them with A_ or 1_

B: define master-apps/app/local/indexes.conf, which will take precedence over any default/indexes.conf.

Consult the Configuration file precedence documentation to see the full explanation of which app/config takes precedence.

In my opinion it's wrong for 3rd party apps to configure index paths without making use of the environment variable.

[mayurr98]

Hi @Kozanic

well, you have installed Splunk on the /opt and you're using the /opt as well for your indexes. Read about the indexes.conf how to change the homePath for your indexes.

To move your existing data to another location proceed like this:

1) stop Splunk

2) change the homePath in indexes.conf

3)move all existing data from old homePath (in your case /opt/splunkhot/_audit/db) to new homePath

4)start Splunk

In a cluster environment you should do this on cluster master and then push indexes.conf to all indexers.
I hope this helps .

After moving the data you may have to clean eventdata :

1)Stop Splunk. Erase all your logs under $SPLUNK_HOME/var/log/splunk

2)Open Command Line and cd to the $SPLUNK_HOME/bin directory.

3)Type ./splunk clean eventdata

4)Enter your splunk admin / password

5)Start Splunk

I hope this helps!

[Kozanic]

Thanks for the help mayurr98.

I have checked indexes.conf - homePath on all indexes is set to $SPLUNK_DB/IndexName/db/.

As mentioned, I have updated $SPLUNK_DB via splunk-launch.conf on both indexes in my cluster - yet I'm still seeing some indexes that are using the old path.

When running | dbinspect index=yourindex I get the below. seems to indicate that internal indexes are holding the old path where as others are picking up the new path.
_telemetry - /opt/splunkhot/_telemetry/db/db_1515589234
_introspection - /opt/splunkhot/_introspection/db/rb_1515619431
dlm_uberagent_log - /var/lib/db/splunkhot/dlm_uberAgent_log/db/rb_1515619434
dlm_uberagent - /var/lib/db/splunkhot/dlm_uberagent/db/rb_1515575095

Not sure why the internals are holding the old path.

Is there another spot I need to check for config??

[mayurr98]

Are you forwarding the internal events of the master to indexers? (recommended). When you did that, only have to have care about the location in the indexers.

You have to configure in the indexes.conf a new location for _audit _internal _introspection and the best way is using volume definition.

Do this in a the master-apps/_cluster/local/indexes.conf

Certainly copy the default stanza to local/indexes.conf and change the home path

[micahkemp]

You may try clearing the messages to see if they return. I've seen instances where the messages will not clear on their own.

[Kozanic]

Messages return after clearing.

I also see them when I run the health check from Monitoring Console

[HiroshiSatoh]

If you change the following settings, the error disappears. Restart is necessary.
But is it correct that the capacity of the disk is 5 G or less?

settings » Server settings » General settings
Pause indexing if free disk space (in MB) falls below *

[Kozanic]

HI HiroshiSatoh,

I did see this setting, unfortunately I cannot update it as the current space being used is that shared with the OS and total disk space is down to 5Gb free - hence the attempts to move the indexes.

[HiroshiSatoh]

Looking at the message, SPLUNK_DB is not "/var/lib/db/splunkhot", but is the setting OK?
Have you restarted?

Data writes to index path '/opt/splunkhot/_audit/db'cannot safely proceed.

[Kozanic]

A few times