- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Splunk Community,
I am running Splunk Enterprise Version: 9.2.3
Steps to reproduce:
- Make a config change to an app on the Cluster Manager - $SPLUNK_HOME/etc/master-apps/<custom_app>/local/indexes.conf
- Validate and Check Restart from Cluster Manager GUI.
- Bundle Information:
- Updated Time shows a date/time from last month (did not update)
- The Active Bundle ID did not change
Unable to make changes to apps and have them pushed to Indexers.
Note: there are other issues
- All three of my clustered Indexers are in Automatic Detention
- Seeing these Messages on GUI:
Search peer xxx has the following message: The minimum free disk space (1000MB) reached for /opt/splunk/var/run/splunk/dispatch.
Search peer xxx has the following message: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly
Ultimatley, I am trying to push changes to the setting frozenTimePeriodInSecs to reduce stored logs and free up space. Thanks for your help
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![SplunkTrust SplunkTrust](/html/@E48BE65924041B382F8C3220FF058B38/rank_icons/splunk-trust-16.png)
Hi
those error messages means that you haven't enough space on indexers as you already know and which you try to fix. Probably you have even so less free space that CM cannot push those new bundles into search peers?
You must log into those nodes or use other tools which can check the disk space situation on all those nodes. It's quite possible that you must manually delete/move some stuff away from those disk partitions to apply a new cluster bundle. But it's hard to say before we know the real situation on those search peers.
btw. have you also try to apply that cluster bundle on GUI or just validate it?
r. Ismo
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![SplunkTrust SplunkTrust](/html/@E48BE65924041B382F8C3220FF058B38/rank_icons/splunk-trust-16.png)
Hi
those error messages means that you haven't enough space on indexers as you already know and which you try to fix. Probably you have even so less free space that CM cannot push those new bundles into search peers?
You must log into those nodes or use other tools which can check the disk space situation on all those nodes. It's quite possible that you must manually delete/move some stuff away from those disk partitions to apply a new cluster bundle. But it's hard to say before we know the real situation on those search peers.
btw. have you also try to apply that cluster bundle on GUI or just validate it?
r. Ismo
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello isoutamo,
Thanks for your help! I was able to log into one of the indexers and manually set frozenTimePeriodInSecs to a lower value. This seemed to then allow me to Validate and Check, and then Push the new bundle from the Cluster Manager.
So, it seems things are much more stable and the errors and warnings have disappeared. But my indexers are still showing about 94% full for the /opt/splunk folder.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![SplunkTrust SplunkTrust](/html/@E48BE65924041B382F8C3220FF058B38/rank_icons/splunk-trust-16.png)
![](/skins/images/FE4825B2128CA5F641629E007E333890/responsive_peak/images/icon_anonymous_message.png)