I created an alert that will email me any errors that come from my various scripted inputs. The search itself seems to work fine when I run it interactively, so I scheduled it. But when I get the email alert it seems to be truncated in a very odd way. At first I though it was my blackberry refusing to download the entire message, but I've ruled that out...
The search: (reformatted for readability)
index=_internal sourcetype=splunkd component=ExecProcessor "message from" NOT (splunk-regmon OR splunk-wmi)
| rex " - message from \"(?<script>[^\"]+)\""
| rex mode=sed "s/^.* - message from \"[^\"]+\" (.*)$/\1/"
| transaction fields="host,source,script" maxpause=45s
| dedup host, script, _raw
| fields host, script
My email alert looks like this:
To:
[email protected]
From:
[email protected]
Subject:[Splunk Alert] Splunk Alert - Scripted input errors
Saved search results.
Name: 'Splunk Alert - Scripted input errors'
Query Terms: 'index=_internal sourcetype=splunkd component=ExecProcessor "message from" NOT (splunk-regmon OR splunk-wmi) | rex " - message from \"(?
Notice that the search is truncated and that I'm missing the actual search results.
Any ideas?
I'm running Splunk 4.1.5, on Ubuntu 8.04 (32 bit), with the postfix mailer, and HTML results.
Background:
I recently had a situation where I lost a bunch of events because of some stupid typo in one of my scripted inputs, so I figured I should setup an alert to avoid this in the future. I found out that any messages that your script writes to stderr (the "standard error" output stream) will be logged in splunk's internal logs with the ExecProcessor component. (I'm using a python input script; and python writes any unhanded exceptions to stderr by default, just like most programming languages). So from there it was simply a matter of filtering out errors created by Splunk's built-in scripted inputs, since I don't care about those errors, so that I can get an email with just the stuff I want to see.
... View more