topic Correlate field from earlier event with error in Splunk Search

Correlate field from earlier event with error

mfrost8 — Wed, 26 Sep 2012 19:39:14 GMT

I have a user who has a need to look for exceptions in a log file, and then figure out what order the error pertained to by looking for the most recently indexed previous event that defined a particular field.

That is, the exception itself won't say anything about the order that caused it, but then again, the line that has the order on it doesn't show the exception (or any error, really). This means I have to somehow tell Splunk to look backwards to find the data (field) that I really need to report on.

How can I do this rather odd request?

thanks

Re: Correlate field from earlier event with error

kristian_kolb — Wed, 26 Sep 2012 21:02:51 GMT

You'd be able to get better help if you include some sample events. It's a bit unclear (to me) how the events are linked together...

Re: Correlate field from earlier event with error

emiller42 — Thu, 27 Sep 2012 00:49:40 GMT

streamstats might actually help here. Add the following to your search (Change fields as needed)

| streamstats last(orderNumber) as last_order

This will create a new field called last_order for each event which will have the last seen orderNumber up to the point of that event occurring.

So something like this:

_raw=orderNumber=123       last_order=123
_raw=some other message    last_order=123
_raw=orderNumber=456       last_order=456
_raw=another message       last_order=456
_raw=ERROR Message!        last_order=456

Which will associate the last seen order number with the error message even though there's no explicit correlation.

(Note, you can add a 'by' clause to the streamstats if you need to restrict the aggregation somehow, say by host)

Streamstats Documentation

Re: Correlate field from earlier event with error

mfrost8 — Thu, 27 Sep 2012 14:19:08 GMT

I'd thought about that, but there's not much to show. These are really horrible, user-generated "logs". Picture what is effectively a dump of 1 long row from a database (no values) followed by a line or two later (no datestamps!) a java exception.

Re: Correlate field from earlier event with error

mfrost8 — Thu, 27 Sep 2012 17:52:13 GMT

Wow. Very nice. I looked at the streamstats command based on a previous post, but I found the fact that it said it was like stats to be confusing as I didn't want any kind of cumulative stats done on the events.

That was what I was looking for. Thanks.