Timestamp/regex help

[a212830]

Hi,

I'm having a hard time getting a regex/timestamp to work properly.

Here are some sample events:

[5892][9008][03/26/2016][11:00:36.607][][][][CServer::ProcessRequest][CServer.cpp:5901][][Leave function CServer::ProcessRequest][][][][][][][][][712][][][][][]
[5892][9008][03/26/2016][11:00:36.607][][][][CSm_Az_Message::ProcessMessage][Sm_Az_Message.cpp:371][][Leave function CSm_Az_Message::ProcessMessage][][][][][][][][][712][][][][][]
[5892][9008][03/26/2016][11:00:36.607][][][][CSm_Az_Message::IsAuthorized][IsAuthorized.cpp:877][][Leave function CSm_Az_Message::IsAuthorized][][][][][][][][][true][][][][][]
[5892][9008][03/26/2016][11:00:36.607][][][][CSm_Az_Message::SendReply][Sm_Az_Message.cpp:575][][Leave function CSm_Az_Message::SendReply][][][][][][][][][][][][][][]
[5892][9008][03/26/2016][11:00:36.607][default_ei1059vwin][s13435/r6559][][CSm_Az_Message::ProcessMessage][Sm_Az_Message.cpp:571][][** Status: Authorized. ][][][][][][][][][][][][][][]

Here's my props:

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\[\d+\]\[\d+
NO_BINARY_CHECK=true
disabled=false
TZ=America/New_York
TIME_PREFIX = ^\[\d+\]\[\d+\]\[
TIME_FORMAT=%m/%d/%Y][%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=75

I get an error: "could not use strptime to parse timestamp...make sure a prefix pattern is specified."

[somesoni2]

The following worked fine for me

SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)\[\d+\]\[\d+
TIME_PREFIX=^\[\d+\]\[\d+\]\[
TIME_FORMAT=%m/%d/%Y][%H:%M:%S.%N
MAX_TIMESTAMP_LOOKAHEAD=24
 TZ=America/New_York

The attribute MAX_TIMESTAMP_LOOKAHEAD is basically length of the timestamp value. It how far splunk should look after the TIME_PREFIX not the how far it is from the start of the event.

[landen99]

This looks just like mine with a few very small differences. I think that the big takeaway was escaping the brackets in the regex. If a good answer already exists, it makes sense to simply add a comment to the answer with any small changes or recommendation. No need for anyone to read through a dozen answers saying exactly the same thing.

[n00badmin]

Your events are already single line events, so to remove debug of the line breaker, I explicitly set break each line and was able to extract the timestamp with:

Update: added landen99's line breaker suggestion below in case you absolutely have to have it and this props still working for me....

[ splunk_answers_timestamp ]
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
BREAK_ONLY_BEFORE = ^\[\d+\]\[\d+
TIME_FORMAT = %m/%d/%Y][%H:%M:%S.%3N
TIME_PREFIX = ^\[\d+\]\[\d+\]\[
TZ = America/New_York
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD=75

I think perhaps you were't accounting for what chars would be left after your line breaker and time prefix....also not sure if you escaped everything you needed to...

[landen99]

I expect that this should work:

TZ=America/New_York
TIME_FORMAT=%m/%d/%Y][%H:%M:%S.%3N

You only need those two lines. If it doesn't work, try escaping the brackets and let us know.

Added:
If you want to add the other lines, these ones are default and are not required/useful:

SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
disabled=false

For the other lines, I would suggest (the brackets need to be escaped):

LINE_BREAKER=^\[\d+\]\[\d+
TIME_PREFIX =^(\[\d+\]){2}\[
MAX_TIMESTAMP_LOOKAHEAD=75
KV_MODE = none

[a212830]

We purposely don't use auto linebreaking or auto timestamp, to improve performance. Every props must have a LINE_BREAK, TIME_STAMP, TIME_PREFIX, KV_MODE, ANNOTATE_PUNCT....

[mattymo]

When I wrote these events to file, the auto linebreak and auto timestamp extract worked perfectly fine...any reason you are using a custom line break??