I have a subset of servers that all of their logs parse the timestamps incorrectly at 12 (noon)..
sample log lines:
CORRECTLY PARSED:
The following all index into splunk as 12PM
May 14 12:00:31 dnv-scan001.scanalert.com resin_stdout: : [12:00:31.153] [SCAN_QUEUE ][WARN ]SaScanQueueServlet: Finished (4ms)
May 14 12:00:56 dnv-scan001.scanalert.com resin_stdout: : [12:00:56.492] [MEMORY ][WARN ]AppMem=25295248 TotalMem=62128128
May 14 12:00:56 dnv-scan001.scanalert.com resin_stdout: : [12:00:56.494] [SCAN_SEND ][WARN ]SaCronSendCompletedServlet: Finished (342ms)
PARSED INCORRECTLY
The following all index into splunk as 12AM
May 14 12:00:02 am1-scan001.scanalert.com resin_stdout: : [12:00:01.605] [MEMORY ][WARN ]AppMem=26510144 TotalMem=69664768
May 14 12:00:31 am1-scan001.scanalert.com resin_stdout: : [12:00:31.624] [SCAN_QUEUE ][WARN ]SaScanQueueServlet: Finished (7ms)
May 14 12:00:51 am1-scan001.scanalert.com resin_stdout: : [12:00:51.622] [SCAN_SEND ][WARN ]SaCronSendCompletedServlet: Finished (0ms)
I have a subset of 17 servers, that parse incorrectly, the ONLY difference I can find is the hostnames.
has anyone encountered this behavior, and have any idea if there is a fix around this?
All logs are in 24HR format fo "midnight" hour shows as 00:00:00, while "noon" as 12:00:00
This turned out to be a simple oversight. It seems that the parser was picking up the "am" from the begining of my hostname in the next field. Hence only happening on machines names "am....."
Fix =
Add the following to my stanza's in props.conf
MAX_TIMESTAMP_LOOKAHEAD = 15