I messed around for several hours trying to make this work, but couldn't get what I was looking for. There are a couple of issues with this. One, upstart wants to directly launch the daemon processes, which doesn't play well with the "splunk start" command which really a wrapper that kicks off several background processes ( splunkd and splunkweb ). So the options basically come down to (1) launching the all the various splunk daemons independently with separate upstart configurations, or (2) using upstart as a lame wrapper and simply sticking "splunks start" and "splunk stop" as pre-start and post-stop scripts at which point upstart wouldn't even know the currently status of the process. In older upstart versions you could manually specify a PID file, but that's gone away quite some time ago. The problem with #1 is splunkd. Launching splunk web (which is really just cherrypy, which is really python) wouldn't be to bad, but launching splunkd manually would probably take some additional understanding of what the "splunk start" process does before actually launching the "splunkd" process, and chances are that could change a bit between versions of splunk, so even if you straced the whole thing and converted it into a nice little pre-start script in a upstart job, it could change next time you upgrade splunk, and some critical startup tasks could be missed. Which could be a big deal. I stuck with the default (or mostly default) init.d script myself, but if anyone comes up with something better, I'd like to know about it too. ... View more

Note: You also have a minor typo in your source stanza: [source::///var/log/snmptraps.log] Should be: [source::/var/log/snmptraps.log] The triple slash extra two slashes are only used in [monitor://...] segments. ... View more

Look at the hidden _indextime field. I've found it helpful to look at the delay of each event and chart the results like so: your search ... | eval delay=_indextime-_time | bucket bins=30 delay | chart count by delay ... View more

I think the issue is with your "where" clause: ... | where EUORCABS=eu This is evaluating the values of fields EUORCABS and eu. That's not what you want to do. Either use ... | where EUORCABS="eu" or ... | search EUORCABS=eu ... View more

The fieldformat search command isn't supported until Splunk 4.2. ... View more

Is any of the stuff on this page helpful? http://splunk-base.splunk.com/answers/4075/whats-the-best-way-to-track-down-propsconf-problems ... View more

I think you simply want [mysourcetype] LINE_BREAKER = (~\\) # You may need to increase this (default 100) LINE_BREAKER_LOOKBEHIND = 1000 SHOULD_LINEMERGE = false There are two things to consider here: 1.) Splunk wants a matching group in the LINE_BREAKER, and 2.) I'm not sure it's valid to end a regex with the backslash ( \ ) character. But I could be wrong. I just re-read the question, and it sounds like you also want newlines to be split events. If that's correct, then try the following: LINE_BREAKER = (~\\|[\r\n]+) ... View more

Hmm, can you use ^ in LINE_BREAKER ? I would think that you'd always need to use something like [\r\n]+ instead of ^ or $ ... Just my 2 cents.. And after re-reading all this info, I don't think you want to use end-of-string ( $ ), start-of-string ( ^ ), or traditional-end-of-line ( [\r\n] ) stuff at all... ... View more

I updated the formatting in this answer and perhaps tweaked regex as well. There seemed to be some extra " \ s" in there which sometimes happens with formatting on this site. I'd double check what you have and maybe try this again. This approach should take all event-breaking logic out of the picture by forcing splunk to use custom line-breaking logic instead of event-combining logic which is where you were having issues. (This should also be faster. The downside is that it's more rigid and could cause issues if your log file varies much.) ... View more

BTW, %+ gives a timestamp like: Sat Oct 8 06:41:52 EDT 2011 , so %s is certainly what you want here. (Just a side note, I was curious about this. This certainly doesn't solve your problem, unfortunately.) ... View more

Good question 😉 Normally, I use chart if I'm looking for a visual display, and the stats search command for anything else. But they do seem fairly interchangeably (certainly all the stats-related command share common functions, which is nice.) Perhaps there's a legacy reason for having both. I'm looking forward to an informed answer. ... View more

I think your TIME_FORMAT should be " %s " and not " %+ "; I've never heard of " %+ " before, but it's possible I'm missing something. Of course, that doesn't solve your breaking issue. ... View more

I'm not 100% sure, but here's something to try: BREAK_ONLY_BEFORE = ^#\s+\w+\s+\d+\s+ There really isn't any value in trying to match the entire line. And on top of that, trying to use $ could be your problem. In regex terms, depending on the mode, $ can mean end-of-line or end-of-string, and I'm really not sure if this regex is applied line-by-line or not. I just know that I've gotten into trouble before with assumptions about ^ and $ , and that could be having an effect here too. In this case, you certainly don't need the extra " .*$ " on the end, so I'd drop it in any case. If that doesn't work, try also replacing the ^ (for the same reason). BREAK_ONLY_BEFORE = [\r\n]+#\s+\w+\s+\d+\s+ BTW, is there any reason not to put the word "modify" in there instead of \w+ ? Also, I assume you realize that only newly indexed events will be effected by this index-time change. It wouldn't hurt to add a TIME_PREFIX and TIME_FORMAT in there too, since there seem to be multiple unix epoch time sprinkled in your example. ... View more

Assuming that the error log is being indexed by Splunk and that the timestamps are correct, then you really should be able to just search for the time range of the last day and see if any events were created for a "source" pattern that matches your log files. For example, lets say your log is called: /var/log/myapp-error.2011-10-07 Then this simple search should do the trick: source=/var/log/myapp-error* earliest=-1d@d But if you need to actually check on the timestamp within the file name for some reason (like your log doesn't have properly timestamps, or timestamps in the past), then should be able to extract out the date from the "source" (log file name), and do something like this: source=/var/log/myapp-error* | rex field=source "error-(?<log_date>\d{4}-\d{2}\d{2})" | eval todays_date=strftime(time(), "%Y-%m-%d") | where log_date=todays_date ... View more

You may have found a known bug. Check out the following post and see if this workaround works for you: http://splunk-base.splunk.com/answers/27748/deployment-clients-phonehomeintervalinsecs-setting-is-not-being-honored-according-to-splunkd_accesslog ... View more

Note that the default was bumped to max_searches_per_cpu=4 in 4.1 (or possibly earlier). So the "Defaults to 2" is no longer accurate. I had bumped this to "3" years ago in my local/limits.conf , so after and upgrade I end up with a lower value than the default. Just a heads up! ... View more

I was actually talking about 4.1, before that setting existed. (My understanding was that HEADER_MODE=always was the default in 4.1 and earlier.) ... View more

The link in this answer no longer works. But the wiki link in the question works again... FYI ... View more