Validate and Check Restart Does Not Update Bundle

[SplunkNinja]

Hello Splunk Community,

I am running Splunk Enterprise Version: 9.2.3

Steps to reproduce:

• Make a config change to an app on the Cluster Manager - $SPLUNK_HOME/etc/master-apps/<custom_app>/local/indexes.conf
• Validate and Check Restart from Cluster Manager GUI.
• Bundle Information:
  • Updated Time shows a date/time from last month (did not update)
  • The Active Bundle ID did not change

Unable to make changes to apps and have them pushed to Indexers.

Note: there are other issues

• All three of my clustered Indexers are in Automatic Detention
• Seeing these Messages on GUI:

Search peer xxx has the following message: The minimum free disk space (1000MB) reached for /opt/splunk/var/run/splunk/dispatch.

Search peer xxx has the following message: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly

Ultimatley, I am trying to push changes to the setting frozenTimePeriodInSecs to reduce stored logs and free up space.  Thanks for your help

[isoutamo · Accepted solution]

Hi

those error messages means that you haven't enough space on indexers as you already know and which you try to fix. Probably you have even so less free space that CM cannot push those new bundles into search peers? 

You must log into those nodes or use other tools which can check the disk space situation on all those nodes. It's quite possible that you must manually delete/move some stuff away from those disk partitions to apply a new cluster bundle. But it's hard to say before we know the real situation on those search peers.

btw. have you also try to apply that cluster bundle on GUI or just validate it?

r. Ismo

[SplunkNinja]

Hello isoutamo,

Thanks for your help!  I was able to log into one of the indexers and manually set frozenTimePeriodInSecs to a lower value.  This seemed to then allow me to Validate and Check, and then Push the new bundle from the Cluster Manager.

So, it seems things are much more stable and the errors and warnings have disappeared.  But my indexers are still showing about 94% full for the /opt/splunk folder.

[isoutamo]

Usually you don't keep your indexes on the same filesystem than your splunk binaries and configurations are. Try to add some more disk space (I prefer to use LVM on linux) and start to use splunk volumes. With those your life is much easier. There are many (or at least some) answers where we have discussed those. Also you should read more about those from docs.