Getting Data In

Issue with timestamp extraction

vishaltaneja070
Motivator

Hello

I have used the below setting in props, but the first event is not able to extract the timestamp:

[sourcetype]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%m/%e/%Y %H:%M:%S %p
TIME_PREFIX=[\r\n]+
MAX_TIMESTAMP_LOOKAHEAD=50
LINE_BREAKER=([\r\n]+)(\/\/\s[\-]+\s\/\/)
BREAK_ONLY_BEFORE=([\r\n]+)(\/\/\s[\-]+\s\/\/)
TRUNCATE=0
SEDCMD-EXTRALINES=s/\/\/\s[\-]+\s\/\///g

Event Sample

6/12/2019 4:12:40 AM (6/12/2019 8:12:40 AM) LogLevel=Information
Process_ID=6175, Thread.Thread_ID=8588 Thread.Thread_Name=()
Message=c
Extended Properties
Origin=hxhdgjshjs
TrackingToken=1c1fb75e-jdhhdd-jdhjdhd

// ----------- //
6/12/2019 4:13:40 AM (6/12/2019 8:13:40 AM) LogLevel=Information
Process_ID=6175, Thread.Thread_ID=85883 Thread.Thread_Name=()
Message=c
Extended Properties
Origin=hxhdgjshjs
TrackingToken=1c1fb75e-jdhhdd-jdhjdhd

// ----------- //
6/12/2019 4:14:40 AM (6/12/2019 8:14:40 AM) LogLevel=Information
Process_ID=6175, Thread.Thread_ID=85488 Thread.Thread_Name=()
Message=c
Extended Properties
Origin=hxhdgjshjs
TrackingToken=1c1fb75e-jdhhdd-jdhjdhd

// ----------- //

Also I don't want to use SHOULD_LINEMERGE=true. Thanks!

0 Karma

woodcock
Esteemed Legend

The problem is that the SEDCMD is removing the lines that you are depending on and need for LINE_BREAKER. You do not need the SEDCMD because the LINE_BREAKER will remove them. Also, I am not sure what will happen when your LINE_BREAKER has multiple capture groups so I would use this:

[<Your Sourcetype Here - YOU MUST SET THIS>]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n\s]+\/\/\s+[\-]+\s+\/\/[\r\n\s]*)
NO_BINARY_CHECK=true
TIME_PREFIX=^
TIME_FORMAT=%m/%d/%Y %H:%M:%S %p
MAX_TIMESTAMP_LOOKAHEAD=22
TRUNCATE=999999

If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value, then you must deploy this to the first full instance(s) of Splunk that handles the events (usually either the HF tier, if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

0 Karma

maciep
Champion

I would probably do something like this, which should start each event at the timestamp. of course, you can use sed to get rid of the line separators if you don't want them

[your_sourcetype]
LINE_BREAKER = ([\r\n]+)(?=\d+\/\d+\/\d+\s+\d+:\d+:\d+)
TIME_PREFIX = ^
TIME_FORMAT = %m/%d/%Y %H:%M:%S %p
MAX_TIMESTAMP_LOOKAHEAD = 24
SHOULD_LINEMERGE = false
0 Karma

493669
Super Champion

Try

TIME_PREFIX=^

since timestamp is at the start of line.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...