Splunk Search

Splunk fails timestamp recognition on almost identical event

hethu
Path Finder

Hi, needs some help with timestamp recognition problem. I have two almost identical events that are sendt over udp to splunk; in one the epoch timestamp is recognized correctly, in the other splunk uses time received. Does anyone have any idea why this is happening?
The events are generated by a powershell script.

hethu_1-1602140947207.png

 

 

Labels (1)
Tags (2)
0 Karma

hethu
Path Finder

Props.conf for the sourcetype:

[sys_multiline]
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 32
NO_BINARY_CHECK = true
REPORT-syslog = syslog-extractions
TIME_FORMAT =
TRANSFORMS = syslog-host
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF
disabled = false
maxDist = 3
pulldown_type = 1
DATETIME_CONFIG = \etc\datetime.xml

Timestamp recognized:

MemoryTimeStamp : 1602139715
MemoryInstanceName : Test-Message
MemoryCookedValue : 0
MemoryUseGb : 0
ComputerName : PC01

From powershell:

<134> 10.0.0.10:

MemoryTimeStamp : 1602139715
MemoryInstanceName : Test-Message
MemoryCookedValue : 0
MemoryUseGb : 0
ComputerName : PC01




Timstamp not recognized:

Oct 8 08:40:41 10.0.0.10 1602139237 10.0.0.10:
RemMemTimeStamp : 1602139237
MemoryCookedValue : 0
RemainingMemoryGb : 0
TotalPhysicalMemoryGB : 0
ComputerName : PC01

From powershell:

<134>1602139237 10.0.0.10:

RemMemTimeStamp : 1602139237
MemoryCookedValue : 0
RemainingMemoryGb : 0
TotalPhysicalMemoryGB : 0
ComputerName : PC01

0 Karma

gcusello
Legend

Hi @hethu,

in the second sample, you have in the beginning of log something that Splunk thinks to be a timestamp, you have to exactly identify timestamp with TIME_PREFIX option in props.conf, something like this:

TIME_PREFIX = RemMemTimeStamp\s+:\s+ 

Ciao.

Giuseppe

 

hethu
Path Finder

Are you referring to the part in red below? this is something that splunk must be inserting by it self on ingestion, this is not part of the message sendt to splunk. I have 3 types of near identical events received on this same sourcetype, on 2 of 3 the timestamp recognition works,  can it really be a wrong regex expression?

All 3 events starts the message with timestamp and ip-address, so this is exactly the same on each one, but 1 of 3 is interpreted differently

Timstamp not recognized:

Oct 8 08:40:41 10.0.0.10 1602139237 10.0.0.10:
RemMemTimeStamp : 1602139237
MemoryCookedValue : 0
RemainingMemoryGb : 0
TotalPhysicalMemoryGB : 0
ComputerName : PC01

Tags (2)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

"All 3 events starts the message with timestamp and ip-address, so this is exactly the same on each one, but 1 of 3 is interpreted differently"

Your examples show that the working one does not have the timestamp at the beginning, it begins with the ip address. This means that the start of the timestamp from MemoryTimeStamp is within the 32 characters defined by MAX_TIMESTAMP_LOOKAHEAD, whereas you would need this set to something more like 42 to find the timestamp in the failing case. Having said that, when it is present at the beginning of the event, I am not sure why it isn't picking it up. 

hethu
Path Finder

Thank you!
Increasing the lookahead  works for timestamp recognition, but it seems like it is this "RemMemTimeStamp : 1602139237" it is recognizing, not the timestamp ahead of it. Would not splunk include it in the event if it was recognized? instead the events start with the remmemtimestamp

Regarding the missing timestamp; it is there...but it seems i used an older image when i posted,  sorry for that.




0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You could try

TIME_FORMAT = %s

Reset lookahead back to 32 to see if this would have picked up the timestamp from the beginning of the line 

0 Karma

gcusello
Legend

Hi @hethu,

as I said, if in your logs there's something that Splunk can recognize as a timestamp (e.g. the red part), to be sure that you take the correct one use the TIME_PREFIX option.

Ciao.

Giuseppe

0 Karma

gcusello
Legend

Hi @hethu,

could you share the props.conf of the wrong recognition and a sample of you logs (both correct and wrong) as text and not as image?

Ciao.

Giuseppe

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!