As I mentioned in a comment above, SEDCMD is evaluated after timestamp extraction, so you can't fix this via transform. You can, however, explicitly tell Splunk what the time format is. (Details on How Indexing Works) props.conf [your_sourcetype] TIME_PREFIX=^ TIME_FORMAT=%m/%d/%I:%M:%S %p MAX_TIMESTAMP_LOOKAHEAD=17 This tells Splunk that the timestamp comes at the beginning of the event (TIME_PREFIX), it has the above strftime format, and it extends, at most, 17 characters into the event. Everything it needs to know to get the timestamp right. This may still not work, as without a year it's not a valid timestamp, so Splunk may still do funny things with it. The real fix is to get your developers to log in a non-ridiculous format. (What Java devs have against ISO standard timestamps, I'll never figure out) I would also set a couple other things: [your_sourcetype] TIME_PREFIX=^ TIME_FORMAT=%m/%d/%I:%M:%S %p MAX_TIMESTAMP_LOOKAHEAD=17 SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+)(?:\d{2}\/\d{2}\/\d{2}:\d{2}:\d{2}) TRUNCATE=999999 This further tells Splunk how to handle the incoming events. Specifically, we're telling Splunk where events begin and end explicitly, so it doesn't have to figure it out. (You'll appreciate this when it stops Splunk from doing bad things with stacktraces) Ideally, you should be setting all of these for every new sourcetype you ingest whenever possible. ... View more

EDIT: New details as of 12/11. Scroll down! Answering this one myself to get the info out there: This has been registered as a bug with Splunk support. (SPL-109514) Details of the issue, how to detect it, and how to work around it follow: Description There appears to be a problem with quota calculation in the search scheduler that is specific to a clustered deployment. Splunk will not dispatch a saved search if the user has reached their concurrent search quota. (As defined in authorize.conf) However, it appears the current usage is not calculated correctly, causing the user to show as over-quota. We see this 'usage' value slowly grow over time. Splunk emits WARNs when this happens: 11-18-2015 11:10:34.638 -0600 WARN SHPMaster - Search not executed: Your maximum number of concurrent searches has been reached. usage=41 quota=30 user=someuser. for search: nobody;search;A Saved Search While some of these may be legit, users affected by this bug generate a far higher volume of them. (We see WARNs every 7 seconds for each scheduled search owned by the affected user) A confusing side-effect of this is that because the searches are never dispatched, they don't report as skipped. So if you're looking at scheduler metrics in the DMC, it looks like everything is successfully running. Detection Because of the sheer volume of WARNs generated, you can use that to detect the issue: We run the following with a 5-minute window as an alert: index=_internal sourcetype=splunkd component=SHPMaster "Search not executed: Your maximum number of concurrent searches has been reached" | rex "user\=(?<user>.+)\.\s+for search:\s(?<search_user>[^;]+);(?<search_context>[^;]+);(?<search_name>.+)" | fields _time usage quota user search_* | stats count by user search_name | where count>40 | stats values(search_name) as affected_searches by user Alert if any records are returned. Impact This can prevent alert searches from running. Depending on the importance of those alerts, the impact can be severe. The frequency of this issue can vary, and appears to be related to overall scheduler activity. Our production cluster saw it happen every day or so, while in a lower volume testing environment it could take over a week to surface. Remediation If affected by this issue, a rolling-restart of the search head cluster will get things moving again. However, the issue will recur. So this becomes an active maintenance thing. NEW 12/11 - Workaround This issue is related to new functionality in Splunk 6.3. Pre-6.3, Splunk calculated quotas independently on each search head. In 6.3, this changed to calculating cluster-wide quotas. This new behavior makes sense, but doesn't seem to work correctly in practice. You can restore splunk to it's pre-6.3 behavior by adding the following in limits.conf: [scheduler] shc_role_quota_enforcement = false shc_local_quota_check = true Alternative Workaround A way to prevent the bug from occurring is to remove all role-based concurrent search quotas. Note that this leaves your users free to run concurrent searches up to the sever-based restrictions in limits.conf. Since we weren't certain of the interaction of imported roles here, we explicitly set all roles to zero, including built in roles ('default', 'user', 'power', and 'admin') Example authorize.conf stanza: [role_admin] srchJobsQuota = 0 rtSrchJobsQuota = 0 cumulativeSrchJobsQuota = 0 cumulativeRTSrchJobsQuota = 0 ... View more

Try a non-transaction approach: sourcetype="enable-integration" ("Submitted order" OR "Murex - Received ExecutionReport") | rex "(?i).*?[^=](?P<TAG198>198=[^\x01]*)" | eval start=if(searchmatch("Submitted order"), 1, 0), end=if(searchmatch("Murex - Received ExecutionReport"), 1, 0) | stats earliest(_time) as _time latest(_time) as last_seen sum(start) as started sum(end) as ended by TAG198 | eval time_since_last_seen=now()-last_seen | search ended=0 AND time_since_last_seen>10 Explanation: Search for relevant log events. Extract your TAG198 field Create fields to indicate if the event is a starting or ending event. Useful later. Do a stats for each TAG198, grabbing the earliest and latest timestamps for events, and a sum of the start/end flags. Items which are ended will have a sum(end) > 0 This gives you an easy way to filter for open items. generate a field to show the time difference between when the search is run, and the most recent timestamp of the item filter for items which have not ended, and which have existed for more than 10 seconds. (Adjust as needed) This gives you a means to say "Show me open transactions which started more than X seconds ago" which sounds like what you're looking for. Worth noting: When trying to deal with near-realtime, you're going to start bumping into the potential of false positives caused by the fact that it takes time for data to end up in Splunk. There is a delay between log write > forwarder read > forwarder send > indexer receive > indexer write. Realtime searches insert themselves as far up the pipeline as possible (They're actually injected in the indexing pipeline before most of the parsing) but there's still a non-zero delta between log write > visible search result. When dealing with this stuff, think about how quickly you can realistically action a problem. Getting alerted in realtime is awesome, but pointless if it takes you a couple of minutes to respond to the alert. At that point, being a minute or two behind on detection doesn't hurt your responsiveness, and makes it much easier omit false positives due to timing. This all depends on your actual use case and expectations, so I won't tell you that you don't need realtime. But it is something to really consider. ... View more