I know that Splunk can parse all different types of timestamps, but I've got a funky one. Here's the situation:
AIX has a command called "errpt" which displays logged errors that the system has generated. It looks like this:
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION 8650BE3F 0820122810 I H ent2 ETHERCHANNEL RECOVERY F3846E13 0820122510 P H ent2 ETHERCHANNEL FAILOVER 8650BE3F 0820104410 I H ent2 ETHERCHANNEL RECOVERY F3846E13 0820093810 P H ent2 ETHERCHANNEL FAILOVER 8650BE3F 0820090910 I H ent2 ETHERCHANNEL RECOVERY CB4A951F 0819114610 I S SRC SOFTWARE PROGRAM ERROR CB4A951F 0819114510 I S SRC SOFTWARE PROGRAM ERROR DE3B8540 0817101710 P H hdisk0 PATH HAS FAILED
Thanks to my newly found friend (multikv) I know now what I want to do with this information. I currently have an app that runs the errpt command every few minutes. If a new one appears, it sends us an email. No big deal.
The problem is we end up getting duplicate e-mails, especially if it logs multiple errors over the course of a minute, such as in the example above.
What I'd like to do is parse that timestamp and have Splunk alert us only if new errpt entry has appeared since the previous search. (We run it every 5 minutes).
The timestamp works like this: 0820122810 (08 = month, 20 = day, 12 = hour, 28 = min, 10 = year).
I've tried to tackle this in the past in props.conf using TIME_FORMAT, but it never seems to work correctly. Does anyone have any suggestions?
If you're getting this data out with
TIME_FORMAT won't help you.
TIME_FORMAT only takes effect at index time, and applies to one single event, so if you do configure that correctly (you'd probably need to set BREAK_ONLY_BEFORE_DATE to false, BREAK_ONLY_BEFORE to something new, TIME_PREFIX to something, and maybe MAX_TIMESTAMP_LOOKAHEAD as well) and Splunk picks it up, the entire event will have the timestamp of the first line.
You could make each line have the new timestamp:
... | multikv | rex "^\S+\s+(?<_time>\S+)" | eval _time=strptime(_time,"%m%d%H%M%y")
It might be better to instead change how the data is input in the first place so every event is one line with its own timestamp:
SHOULD_LINEMERGE = false TIME_PREFIX = ^\S+\s+ TIME_FORMAT = %m%d%H%M%y MAX_TIMESTAMP_LOOKAHEAD = 25
And then do an explicit field extraction to replace
EXTRACT-fields = ^(?<ident>\S+)\s+(?<timestamp>\S+)\s+(?<t>\S+)\s+(?<c>\S+)\s+(?<resource>\S+)\s+(?<desc>.*)
Thank you for your response.
Now I'm getting "IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION" as one indexed entry, and the actual error itself as another entry. That wasn't the behavior I was expecting...
If I'm not going to take the multikv approach, I could probably just "sed" out the headers...
I'm supposed to do that on the indexer (not the forwarder), right? Yes, my props.conf is set correctly I believe. But when its indexed, it indexes the time it captures it, not the timestamp of the errpt output. For example, just now:
AA8AB241 0831104210 T O OPERATOR OPERATOR NOTIFICATION
Splunk's timestamp: 10:43:09.000
I have the feeling I'm missing a step somewhere...