I'm sorry to say that I'm running 4.3.3 and I'm still seeing this bug. This is when using Actions>Export Results in a standard search (which uses 'top') The workaround of using sendemail is good enough for now. ... View more

Your final example has one too many 9's in the conditional. Should be index=storm_splunk | eval event_len=len(_raw) | WHERE event_len > 9999 | stats count values(source) values(sourcetype) values(host) values(index) by event_len ... View more

This answer is probably no longer of relevance to you, but perhaps some other reader will stumble upon it as I did. The problem is that you don't have a timestamp in your scripted output, so Splunk goes looking for one. Both values for MaxMemory and TotalMemory appear to Splunk to be valid epoch-relative dates, but I'm guessing that 1908932608, which maps to 29 Jun 2030 03:03:28 GMT is too far in the future. Then it sees 1130823680, which maps to 01 Nov 2005 05:41:20 GMT Every time your script runs, there is likely a slightly different value for TotalMemory so that computed time is going to jump around, as you've experienced. One way around it would be to add an explicit timestamp at the beginning of the line of output. Another would be to disable the timestamp processor for this input so that it would get a timestamp that matched the time the input was received DATETIME_CONFIG=NONE See Disable timestamp processor for more info on the latter. ... View more

Ayn, I combined this with your other answer http://splunk-base.splunk.com/answer_link/41401/ about getting readable times, to get | eval indextime=strftime(_indextime,"%+") Thanks for both excellent answers! ... View more

What about permissions on the /var/log/custom hierarchy? Is it possible that the forwarder is not ingesting logs in there because the splunk user can't read them or search the containing directories? ... View more

OK, so that explains why it SHOULD work in the forwarders' props.conf, but the real problem is that it doesn't. I can see NO_BINARY_CHECK in the output of btool props list for my_sourcetype, but when it's restarted, it still logs 08-14-2012 19:46:58.139 +0000 WARN FileClassifierManager - The file 'my_file' is invalid. Reason: binary 08-14-2012 19:46:58.139 +0000 INFO TailingProcessor - Ignoring file 'my_file' due to: binary for every file. ... View more

We are working this case with support. They've said After some further inquiries with our Dev team, I've learned that the timeout settings in distsearch.conf will not actually have any effect on the problem. It seems that what is happening is that we are timing out at time, while trying to read the auth token from the peer (Unable to connect to peer uri...) . The httpclient timeouts that affect this behavior are actually hardcoded and NOT configurable. connectionTimeout = 5; sendTimeout = 10; rcvTimeout = 10; There isn't one setting exposed which you could use to control such timeouts. ... View more

No, what you mention is about parsing the host from the filename. I was looking to limit what files were ingested, based on matching the hostname in the file. ... View more

Not really, the shared storage itself is multi-node. ... View more

In the end, we changed the config so that the forwarders wrote to local storage. Having Splunk centralizing access to the logs alleviated most of the need to have them on the shared storage anyway. ... View more

Your comment about ${VARIABLE} is the most intriguing. If that works, I'd like to see that documented, although we ended up going a different route. ... View more

For us it was a filesystem issue. Even though the file was constantly being appended to (splunkd.log) the modification timestamp was not being updated. Being the consumate Windows SysAdmins that we are, we rebooted the box and it fixed itself. Fortunately, this was a QA system. ... View more

OK, that would seem to explain it. Now, the only-tangentially-Splunk-related issue is: Why does splunkd.log on my Windows UniversalForwarder not get it's file modification time updated as new log lines are written? ... View more

Mine are also all new 4.2 installs. ... View more

I'm seeing the same thing. What looks really odd to me about the timestamp it complains about, in both yours and mine, is that the year is in an odd location. Your example: A possible timestamp match (Wed Feb 22 17:01:12 2012) is outside of the acceptable time window Having the time splitting up the day/year doesn't look like anything parsed by datetime.xml In my case, the line it is processing is actually from splunkd.log from a windows forwarder, so it's not a timestamp under my control. Nor do I really even see a timestamp of this format in the source file. 03-07-2012 06:34:18.235 +0000 WARN DateParserVerbose - A possible timestamp match (Wed Mar 7 06:34:05 2012) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context="source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log|host::winplay-dc1|splunkd|remoteport::56124" host=lxsplunk-qc1 Options| sourcetype=splunkd Options| source=/opt/splunk/var/log/splunk/splunkd.log Options| splunk_server=lxsplunk-qc1 Options Perhaps this will given someone at Splunk an insight into where to look further. For reference, I'm on 4.2.2 ... View more

Yes, its about not having a single point of failure for the ingestion of the logs, but also not wanting to eat them multiple times. Putting the forwarder on each node allows each node to be responsible for its own logs. It has the added benefit that it allows us to monitor node performance as well. If a node goes down, the remaining nodes continue processing and their logs continue to get consumed by spunk. ... View more

Sorry, neither option is really what I'm looking for. #1 isn't right because it doesn't take into account the differing forwarder hostnames. Separating them out to their own directory isn't really the issue. It's matching the path to the hostname/servername of the forwarder, wherever that happens to be in the monitored filepath. As for #2, I know I could write an external script to modify the inputs.conf, but ideally there is some way to do it from within Splunk, using some kind of variable substitution in the monitor stanza. If it helps, the app will be distributed via deployment server. ... View more

The instructions in rithy's answer above did not quite work for me on a 4.2.2 system. I got it to work with these stanzas: [dashboard_live:_current] TimeRangePicker_0_1_0.default = Last 4 hours [flashtimeline:_current] TimeRangePicker_0_1_0.default = Last 4 hours in the viewstates.conf files previously mentioned: $SPLUNK_HOME/etc/users/ [your user] /search/local/viewstates.conf or $SPLUNK_HOME/etc/apps/search/local/viewstates.conf The minor difference being dashboard becomes dashboard_live and the additional flashtimeline entry. ... View more

While that definitely taught me something about the snap-to (I didn't know you could do math on the right-handside (and I don't see anything in the docs that imply that you can)), it doesn't really get me what I was looking for, which was to snap to the previous half-hour. With the math, I can schedule either the first or last half hour of the previous hour, but I really just want the previous half hour. ... View more