topic Re: AIX errpt timestamp in Getting Data In

AIX errpt timestamp

Branden — Tue, 31 Aug 2010 21:18:54 GMT

I know that Splunk can parse all different types of timestamps, but I've got a funky one. Here's the situation:

AIX has a command called "errpt" which displays logged errors that the system has generated. It looks like this:

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
8650BE3F   0820122810 I H ent2           ETHERCHANNEL RECOVERY
F3846E13   0820122510 P H ent2           ETHERCHANNEL FAILOVER
8650BE3F   0820104410 I H ent2           ETHERCHANNEL RECOVERY
F3846E13   0820093810 P H ent2           ETHERCHANNEL FAILOVER
8650BE3F   0820090910 I H ent2           ETHERCHANNEL RECOVERY
CB4A951F   0819114610 I S SRC            SOFTWARE PROGRAM ERROR
CB4A951F   0819114510 I S SRC            SOFTWARE PROGRAM ERROR
DE3B8540   0817101710 P H hdisk0         PATH HAS FAILED

Thanks to my newly found friend (multikv) I know now what I want to do with this information. I currently have an app that runs the errpt command every few minutes. If a new one appears, it sends us an email. No big deal.

The problem is we end up getting duplicate e-mails, especially if it logs multiple errors over the course of a minute, such as in the example above.

What I'd like to do is parse that timestamp and have Splunk alert us only if new errpt entry has appeared since the previous search. (We run it every 5 minutes).

The timestamp works like this: 0820122810 (08 = month, 20 = day, 12 = hour, 28 = min, 10 = year).

I've tried to tackle this in the past in props.conf using TIME_FORMAT, but it never seems to work correctly. Does anyone have any suggestions?

Thanks!

Re: AIX errpt timestamp

Brian_Osburn — Tue, 31 Aug 2010 21:28:41 GMT

Are you setting the the TIME_FORMAT in props.conf correctly?

[your_sourcetype]

TIME_FORMAT=%m%d%H%M%y

Re: AIX errpt timestamp

Branden — Tue, 31 Aug 2010 21:45:54 GMT

I'm supposed to do that on the indexer (not the forwarder), right? Yes, my props.conf is set correctly I believe. But when its indexed, it indexes the time it captures it, not the timestamp of the errpt output. For example, just now:
AA8AB241 0831104210 T O OPERATOR OPERATOR NOTIFICATION

Splunk's timestamp: 10:43:09.000

I have the feeling I'm missing a step somewhere...

Re: AIX errpt timestamp

gkanapathy — Tue, 31 Aug 2010 21:50:27 GMT

If you're getting this data out with multikv, then TIME_FORMAT won't help you. TIME_FORMAT only takes effect at index time, and applies to one single event, so if you do configure that correctly (you'd probably need to set BREAK_ONLY_BEFORE_DATE to false, BREAK_ONLY_BEFORE to something new, TIME_PREFIX to something, and maybe MAX_TIMESTAMP_LOOKAHEAD as well) and Splunk picks it up, the entire event will have the timestamp of the first line.

You could make each line have the new timestamp:

... | multikv | rex "^\S+\s+(?<_time>\S+)" | eval _time=strptime(_time,"%m%d%H%M%y")

It might be better to instead change how the data is input in the first place so every event is one line with its own timestamp:

SHOULD_LINEMERGE = false
TIME_PREFIX = ^\S+\s+
TIME_FORMAT = %m%d%H%M%y
MAX_TIMESTAMP_LOOKAHEAD = 25

And then do an explicit field extraction to replace multikv:

EXTRACT-fields = ^(?<ident>\S+)\s+(?<timestamp>\S+)\s+(?<t>\S+)\s+(?<c>\S+)\s+(?<resource>\S+)\s+(?<desc>.*)

Re: AIX errpt timestamp

Branden — Tue, 31 Aug 2010 23:04:03 GMT

Thank you for your response.
Hmmm...
Now I'm getting "IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION" as one indexed entry, and the actual error itself as another entry. That wasn't the behavior I was expecting...
If I'm not going to take the multikv approach, I could probably just "sed" out the headers...

Re: AIX errpt timestamp

gkanapathy — Tue, 31 Aug 2010 23:38:16 GMT

Yeah, I skipped out dropping the headers. They don't hurt, and you can transform them out.

Re: AIX errpt timestamp

Branden — Wed, 01 Sep 2010 00:08:28 GMT

All set, working the way I need it to now. Thanks, both of you!