I'm tracking down users that abuse real-time searches, as I've been seeing this gold warning bar a lot lately. Metadata results may be incomplete: 100000 entries have been received from all peers (see parameter maxcount under the [metadata] stanza in limits.conf), and this search will not return metadata information for any more entries. (sid=rt_1380116912.11287.searchhead01) I was surprised that I had three running! I tracked it down to the Search Summary page. I'm assuming the searches update Events Indexed, Earliest Event, and Latest Event. The Jobs page shows one of the searches is: | metadata type=sourcetypes | search totalCount>0 | rename totalCount as Count recentTime as "Last Update" [real-time] The jobs page shows the three are Running (100%), they quickly use 30 MB (and keep climbing, but more slowly), the expiration time always seems to always be 10 minutes in the future. I'd like to make take the real-time out of the search to make it play nice. Is there a way to do this? I've been parked at the summary page for 40 minutes and the searches now use 50 MB. I have pooled search heads and assume this is consuming space in my pool area. My users also get worried when they see the warning messages. I've seen this for version 4 HALP! Consulting the summary dashboard of the search app causes my system to run out of memory! I'm using version 5.0.4, build 172409. ... View more

I realize buckets die off as the newest event surpasses the expiration date. I also understand that deleting events do not remove the events, simply mask them from appearing in search results. My question is, do deleted events count when Splunk decides on when to expire a bucket file? In other words, does deleting an event remove it from Splunk's calculations for expiration? I am looking for a way to manage an index corrupted with future events, other than manually deleting very old files manually, when the time comes. The other events in the index are valid and needed. I am using Splunk version 4.3.4, soon to be upgraded to version 5.x. This is related to my Splunk-Base "How do i configure an index to manage future events" question. An answer here or there may solve both. Please correct me if I misunderstand anything and thanks for the help! ... View more

Our developmeent team is using systems with times set in the future, some as far as two years in the future. I'd like to configure an index for them to use and keep my other indexes realistic, as far as true timestamping goes. Right now I'm running version 4.3.4, with plans on upgrading to version 5 soon. I came up with something like this, ignoring the home/cold/thawed paths: maxDataSize = auto maxHotIdleSecs = 2592000 maxTotalDataSizeMB = 20000 homePath.maxDataSizeMB = 1000 frozenTimePeriodInSecs = 86400 Looking through indexes.conf.specs, the quarantineFutureSecs option looks interesting. How do I search and manage a quarantine bucket? I assume it's not a default option as my main index contains several events more than 30 days in the future. The volume appears to be small, so I'm not too concerned with retaining the index for two years. (My future stance on that may change.) I'd like the buckets to expire very soon after two years. I'm open for suggestions and recommendations. Thanks in advance! ... View more