Hi all ,
Recently we had an issue with /opt as it is consuming 100% memory. We have gone through and checked .bundle files are consuming a large amount of space under this, so we have deleted some files and some recent files which made the /opt get reduced.
Will there be any impact deleting the files? What if we keep a cron job to delete .bundle files as we see many files are getting created recent recently and consuming a lot of memory?
You didn't say which servers you removed the bundles from. I'm assuming its your indexers? The bundle contains knowledge objects from your search head that are necessary for the indexer to perform searches. Generally you should not delete it. Instead, take a look at what is taking up most of the space in the bundle. Often times this will be a large lookup file that may not even be needed. Remove the file (if its not needed) from the originating server (search head). You can also tune your bundle replication to avoid copying large files, see the settings in distsearch.conf:
https://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Distsearchconf
How much space are the bundles consuming? The easier solution may be to expand your /opt partition to handle the bundle size.
You didn't say which servers you removed the bundles from. I'm assuming its your indexers? The bundle contains knowledge objects from your search head that are necessary for the indexer to perform searches. Generally you should not delete it. Instead, take a look at what is taking up most of the space in the bundle. Often times this will be a large lookup file that may not even be needed. Remove the file (if its not needed) from the originating server (search head). You can also tune your bundle replication to avoid copying large files, see the settings in distsearch.conf:
https://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Distsearchconf
How much space are the bundles consuming? The easier solution may be to expand your /opt partition to handle the bundle size.
yes it's in indexers. we have 5 indexers in our environment , we don't what exactly went wrong with customer area , the /opt abruptly started increasing in all the indexers and we encountered search peers and dispatch are the one consuming huge space . we deleted the .bundle files . the whole searchpeers is consuming 20G of space.
The bundles should replicate again, so the problem might re-occur eventually. If this happened suddenly, try and determine what changed recently on your search heads. Take a look at to see if there are any large lookups. I've seen instances where a user generated a multi GB lookup file by using the outputlookup command.