I'm tracking down users that abuse real-time searches, as I've been seeing this gold warning bar a lot lately. Metadata results may be incomplete: 100000 entries have been received from all peers (see parameter maxcount under the [metadata] stanza in limits.conf), and this search will not return metadata information for any more entries. (sid=rt_1380116912.11287.searchhead01) I was surprised that I had three running! I tracked it down to the Search Summary page. I'm assuming the searches update Events Indexed, Earliest Event, and Latest Event. The Jobs page shows one of the searches is: | metadata type=sourcetypes | search totalCount>0 | rename totalCount as Count recentTime as "Last Update" [real-time] The jobs page shows the three are Running (100%), they quickly use 30 MB (and keep climbing, but more slowly), the expiration time always seems to always be 10 minutes in the future. I'd like to make take the real-time out of the search to make it play nice. Is there a way to do this? I've been parked at the summary page for 40 minutes and the searches now use 50 MB. I have pooled search heads and assume this is consuming space in my pool area. My users also get worried when they see the warning messages. I've seen this for version 4 HALP! Consulting the summary dashboard of the search app causes my system to run out of memory! I'm using version 5.0.4, build 172409. ... View more

I realize buckets die off as the newest event surpasses the expiration date. I also understand that deleting events do not remove the events, simply mask them from appearing in search results. My question is, do deleted events count when Splunk decides on when to expire a bucket file? In other words, does deleting an event remove it from Splunk's calculations for expiration? I am looking for a way to manage an index corrupted with future events, other than manually deleting very old files manually, when the time comes. The other events in the index are valid and needed. I am using Splunk version 4.3.4, soon to be upgraded to version 5.x. This is related to my Splunk-Base "How do i configure an index to manage future events" question. An answer here or there may solve both. Please correct me if I misunderstand anything and thanks for the help! ... View more

Our developmeent team is using systems with times set in the future, some as far as two years in the future. I'd like to configure an index for them to use and keep my other indexes realistic, as far as true timestamping goes. Right now I'm running version 4.3.4, with plans on upgrading to version 5 soon. I came up with something like this, ignoring the home/cold/thawed paths: maxDataSize = auto maxHotIdleSecs = 2592000 maxTotalDataSizeMB = 20000 homePath.maxDataSizeMB = 1000 frozenTimePeriodInSecs = 86400 Looking through indexes.conf.specs, the quarantineFutureSecs option looks interesting. How do I search and manage a quarantine bucket? I assume it's not a default option as my main index contains several events more than 30 days in the future. The volume appears to be small, so I'm not too concerned with retaining the index for two years. (My future stance on that may change.) I'd like the buckets to expire very soon after two years. I'm open for suggestions and recommendations. Thanks in advance! ... View more

Looking to recover some disk space. Can I delete old diag-temp directories, anything under run/splunk/diag-temp? Here are some samples I've pasted from tar(1) output. ./run/splunk/diag-temp/diag-hostname-2010-11-23/etc/ngram-models/_hebrew-ISO-8859-8.ngrams, 1244 bytes, 3 tape blocks x ./var/run/splunk/diag-temp/diag-hostname-2010-11-23/etc/ngram-models/_bosnian.ngrams, 1247 bytes, 3 tape blocks x ./var/run/splunk/diag-temp/diag-hostname-2010-11-23/etc/ngram-models/_slovak-CP1250.ngrams, 1215 bytes, 3 tape blocks x ./var/run/splunk/diag-temp/diag-hostname-2010-11-23/etc/ngram-models/_sanskrit.ngrams, 1207 bytes, 3 tape blocks x ./var/run/splunk/diag-temp/diag-hostname-2010-11-23/etc/ngram-models/_hindi.ngrams, 1244 bytes, 3 tape blocks x ./var/run/splunk/diag-temp/diag-hostname-2010-11-23/etc/ngram-models/_catalan.ngrams, 1273 bytes, 3 tape blocks I believe the answer is, "Those are not needed beyond troubleshooting and can be removed," but would like confirmation. Thanks in advance! ... View more

I've recently brought up one additional pooled search head to join my original two. All my search head are version 4.3.4, build 136012. Splunk e-mail alerts coming from the new search head have the format "From: [email protected]", while the other two show "From: Splunk Daemon User [[email protected]]". (Or '@searchhead02.' depending on source. You get the idea.) I do not like localhost@localdomain in the header. I would rather have the more informative, real name included in the mail header. My users agree with me, for once. 😉 I've looked through the GUI and not found differences in Manager > System Setting > {General Setting or Email alert settings}. I've run "find . -type f -exec grep localdomain {} ; -print" and nothing obvious has jumped out at me. I'll admit I haven't compared everything in all the files, however. All three have the same content in .../etc/system/local/alert_actions.conf (Non-pooled directory, no such file in the pooled area). Additionally, I brought up a new search head, that is not pooled, at the same time as the new pooled search head. (I have a total of four search heads.) Same version and build. It also uses the unpreferred "From: [email protected]". Same settings and version as the new pooled search head. It is just not pooled. All are RHEL 5.7. All respond correctly to the hostname(1). hosts(5) files are correct. /etc/sysconfig/network have "HOSTNAME=" set correctly. I've used mailx(P) on all four to send test messages, from the Splunk user, using the command line. The headers all come through with the preferred format of "From: Splunk Daemon User [[email protected]]" so I'm convinced it's not sendmail(8) or the operating system causing the problem. I am not convinced some interaction between the OS and Splunk couldn't cause the problem, however. But I really believe I've missed something in the Splunk configuration on the new boxes. What am I missing? ... View more

I have just a few search heads to manage. (version 4.3.4) A few are pooled, one is not. I'd like to exclude indexes from certain roles. Say this is the desired situation. Power users cannot search/access the accounting and legal indexes. Legal role can search legal indexes, but not accounting indexes. Accounting role can search accounting indexes, but not legal indexes. Working through the GUI Splunk > Manager > Access controls > Roles I find the Search restrictions and Indexes sections where I can restrict or grant a role access to an index. I've been using Indexes section to exclude an index from a role, but the logic is biased towards listing all allowed indexes. If I add an index, I have to circle back and verify everyone has access to the new index. Can I do something like this in the Search restrictions section and completely ignore the Indexes section? In my power user role, enter: index=* NOT (index=legal OR index=accounting) In my legal role, which inherits power, enter: index=* NOT index=accounting in my accounting role, which inherits power, enter: index=* NOT index=legal In my case it works well if I can configure by listing the restricted indexes instead of listing the allowed ones. If I have to use both the Indexes and Search restriction sections, say to also configure access to the internal indexes, how do they interact? Does one section override the other? Do the rules get merged? (If so, how?) Should I avoid using both sections at all costs? Are there hidden costs to using these strategies? ... View more