Hi ALL. Currently I am facing another problem in our distributed environment. We have 5 individual indexer instance configured to store/retrieve data out of which one of the indexer is used almost 87% of disk usage, when compared to other four indexer instance. We have two heavy forwarder and some 1000 + universal forwarders, forwarding the inputs to these five indexer instances. I have checked the outputs.conf, and indexes.con file in both the heavy forwarder and five indexer instance and all share the same configuration details. Splunk Version - 6.2.1 outputs.conf stanza [tcpout] defaultGroup = all_indexers maxQueueSize = 1GB [tcpout:all_indexers] server = splunk01.xxxxx.com:9997,splunk02.xxxxx.com:9997,splunk03.xxxxx.com:9997,splunk04.xxxxx.com:9997,splunk05.xxxxx.com:9997 autoLB = true Kindly let me know how to figure out where exact the problem is and also what to check in the splunkd.log. Thanks in advance. ... View more

Hi All, Currently we could License warning message popping out in splunk web portal ? This is the firs warning message we have got for this month and as per splunk document we are allowed for five violation in 30 day period. Similarly when checked in the license manager under Enterprise license group --> alerts -- > Current & Permanent Current : 1 pool warning reported by 1 indexer Permanent : 10 license window warnings reported by 8 indexers Volume used today is under the threshold limit, but still we are getting this warning message Effective daily volume is 348 GB (License limit) auto_generated_pool_enterprise is 187 GB When checked Usage report Today's License Usage (GB) we could see the license usage is increasing constantly. Kindly let me know how to troubleshoot this issue and control the license before it goes out of hands. And also let me know how to figure out which source/source type/host are consuming more licenses. Details : Splunk 6.2.1 version License type : Enterprise License. thanks in advance ... View more

Hi All, Currently we facing a storage issue in one of the indexer instances, though the retention policy has set for an year and it works for most of the indexes and only for few of the indexes we could see the earliest event date Dec 13, 2013 9:44:29 PM latest event date Mar 21, 2017 10:30:02 AM . We have executed the splunk btool command but not sure what exactly we need to search in that ? we have checked the splunkd.log but unable to find anything related to this index and also we have executed the dbinspect search command with the particular index with time frame set as last 7 days and got some details but not sure what we need to validate from the result. When checked the buckets hotdb, colddb, datamodel_summary, summary thawed db, We could see very huge amount of data being stored in hot and cold buckets. And also I could see the below db in both Hot and cold buckets and their size was in KB. drwx--x--x 3 splunk splunk 4096 Oct 21 2014 inflight-db_1406595487_1406595487_10374 drwx--x--x 3 splunk splunk 4096 Oct 22 2014 inflight-db_1408598841_1408598841_10381 Below are the settings configured in indexes.conf file with retention policy set to Global. Splunk version 6.2.1 [volume:Hot] path = /slogs [volume:Cold] path = /slogs [volume:Base] path = /slogs [default] frozenTimePeriodInSecs = 31536000 enableTsidxReduction = true timePeriodInSecBeforeTsidxReduction = 7884000 Kindly guide me how to fix this issue, as we are running short of disk space. thanks in advance. ... View more