Return ExitCode Value in Alert/Event

[confma812]

Currently monitoring some Windows and Application services via WMI:Service.

I have a list of known Windows Exit Codes for windows processes/services and I'd like to correlate the ExitCode to an Event or Alert when it is reported to users (because there are far too many codes and I don't know every instance that may be generated in the future).

So let's say a service halts and my exit code is 1067, which means "The process terminated unexpectedly." I would like that value to be appended to my alert given the value of my exit code.

Is there a more elegant solution than simply writing a script to look at the values of the alert (for when the service stops)? If not - do Splunk scripts support a stdin/stout like interface where I can return a value to the alert?

[lguinn2 · Accepted solution]

This should be pretty easy in Splunk - I don't think you need a script at all. You are monitoring WMI:Service using Splunk. Therefore I assume that you have a field in the events named EventID or ExitCode. You could create a lookup table of the codes and then include that information as part of your search/alert.

sourcetype=WMI:Service yourerrrortesthere 
| lookup exitcodeLookup ExitCode OUTPUT ExitMessage
| table ExitCode ExitMessage other_stuff_you_want_to_show

There are lots of ways to do this. This example assumes that you have loaded a .csv file for the lookup like this

ExitCode,ExitMessage
100,Bad Input
200,User Error
302,Unknown Result

If you are talking about actual Windows Event Codes, there is a free Splunk app that will look them up for you: http://apps.splunk.com/app/411/

Here is a tutorial on lookups: Add lookup files into Splunk

Or download the free Splunk book: Exploring Splunk

[lguinn2]

You can use the lookup command in any search string. If you make the lookups automatic, you will not even need to specify the command - the fields from the lookup will always be available.

[confma812]

Oh excellent, I am always learning more on the features of Splunk.

With regards to the Lookups as well as the app you suggested, will I be able to specify lookups only for when alerts are triggered, or in that case does it happen for every WMI:Service update interval?