topic Issue with timestamp extraction in Getting Data In

Issue with timestamp extraction

vishaltaneja070 — Fri, 12 Jul 2019 10:10:34 GMT

Hello

I have used the below setting in props, but the first event is not able to extract the timestamp:

[sourcetype]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%m/%e/%Y %H:%M:%S %p
TIME_PREFIX=[\r\n]+
MAX_TIMESTAMP_LOOKAHEAD=50
LINE_BREAKER=([\r\n]+)(\/\/\s[\-]+\s\/\/)
BREAK_ONLY_BEFORE=([\r\n]+)(\/\/\s[\-]+\s\/\/)
TRUNCATE=0
SEDCMD-EXTRALINES=s/\/\/\s[\-]+\s\/\///g

Event Sample

6/12/2019 4:12:40 AM (6/12/2019 8:12:40 AM) LogLevel=Information
Process_ID=6175, Thread.Thread_ID=8588 Thread.Thread_Name=()
Message=c
Extended Properties
Origin=hxhdgjshjs
TrackingToken=1c1fb75e-jdhhdd-jdhjdhd

// ----------- //
6/12/2019 4:13:40 AM (6/12/2019 8:13:40 AM) LogLevel=Information
Process_ID=6175, Thread.Thread_ID=85883 Thread.Thread_Name=()
Message=c
Extended Properties
Origin=hxhdgjshjs
TrackingToken=1c1fb75e-jdhhdd-jdhjdhd

// ----------- //
6/12/2019 4:14:40 AM (6/12/2019 8:14:40 AM) LogLevel=Information
Process_ID=6175, Thread.Thread_ID=85488 Thread.Thread_Name=()
Message=c
Extended Properties
Origin=hxhdgjshjs
TrackingToken=1c1fb75e-jdhhdd-jdhjdhd

// ----------- //

Also I don't want to use SHOULD_LINEMERGE=true. Thanks!

Re: Issue with timestamp extraction

493669 — Fri, 12 Jul 2019 10:27:22 GMT

Try

TIME_PREFIX=^

since timestamp is at the start of line.

Re: Issue with timestamp extraction

maciep — Fri, 12 Jul 2019 10:28:30 GMT

I would probably do something like this, which should start each event at the timestamp. of course, you can use sed to get rid of the line separators if you don't want them

[your_sourcetype]
LINE_BREAKER = ([\r\n]+)(?=\d+\/\d+\/\d+\s+\d+:\d+:\d+)
TIME_PREFIX = ^
TIME_FORMAT = %m/%d/%Y %H:%M:%S %p
MAX_TIMESTAMP_LOOKAHEAD = 24
SHOULD_LINEMERGE = false

Re: Issue with timestamp extraction

woodcock — Sat, 13 Jul 2019 17:10:23 GMT

The problem is that the SEDCMD is removing the lines that you are depending on and need for LINE_BREAKER. You do not need the SEDCMD because the LINE_BREAKER will remove them. Also, I am not sure what will happen when your LINE_BREAKER has multiple capture groups so I would use this:

[<Your Sourcetype Here - YOU MUST SET THIS>]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n\s]+\/\/\s+[\-]+\s+\/\/[\r\n\s]*)
NO_BINARY_CHECK=true
TIME_PREFIX=^
TIME_FORMAT=%m/%d/%Y %H:%M:%S %p
MAX_TIMESTAMP_LOOKAHEAD=22
TRUNCATE=999999

If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value, then you must deploy this to the first full instance(s) of Splunk that handles the events (usually either the HF tier, if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.