Upvoting because this resolved a similar issue for me - when we copied XML from an earlier version of Splunk into the dashboard creation page it worked fine, but afterwards when navigating to the page it would not load the dashboard or nav bar. Removing the head and /head tags worked like a charm.
... View more
If your file system is filling up from bundles, you were likely seeing an extremely large $SPLUNK_HOME/var/run/searchpeers. A search head replicates and distributes its knowledge objects to its search peers in the bundles you see in var/run/searchpeers.
Knowledge objects include saved searches, event types, and other entities used in searching across indexes. The search head needs to distribute this material to its search peers so that they can properly execute queries on its behalf. Bundles typically contain a subset of files (configuration files and assets) from $SPLUNK_HOME/etc/system, $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users. The process of distributing knowledge bundles means that peers by default receive nearly the entire contents of the search head's apps. If an app contains large binaries that do not need to be shared with the peers, that could also be a reason for the large bundle sizes.
You can read more specifically on those bundles here:
The best way to mitigate this issue is to reduce the bundle size on the search head itself. This is done with the replication blacklist (just deleting the bundles will only temporarily resolve disk usage problems, as they will get replicated again if they still exist on the SH). The blacklist allows you to limit what is sent to the search peers (indexers) in the knowledge bundle.
We have an entire documentation page on that here:
As mentioned earlier, most bin directories, jar and lookup files do not need to be replicated to search peers, and can be blacklisted in distsearch.conf. For example, on the search heads:
The ellipsis wildcard … recurses through directories and subdirectories to match.
noBinDir = .../bin/*
jarAndLookups = (jar|lookups)
You can then stop Splunk on each indexer (one at a time) and remove the knowledge bundles in $SPLUNK_HOME/var/run/searchpeers and then start Splunk (the entire contents of $SPLUNK_HOME/var/run/searchpeers can be deleted). The search heads will redistribute the new (reduced size) knowledge bundles.
As an FYI, each indexer keeps 5 knowledge bundles per search head.
... View more