topic Re: nullQueue and messages about problems with timestamp extraction in Splunk Enterprise

nullQueue and messages about problems with timestamp extraction

NoSpaces — Wed, 31 Jan 2024 13:12:01 GMT

Hello to everyone!
One of the source types contains messages with no timestamp

<172>hostname: -Traceback: 0x138fc51 0x13928fa 0x1399b28 0x1327c33 0x3ba6c07dff 0x7fba45b0339d

To resolve this problem, I created a transform rule that successfully eliminated this "junk" from index

[wlc_syslog_rt0] REGEX = ^<\d+>.*?:\s-Traceback:\s+ DEST_KEY = queue FORMAT = nullQueue

But after it, I still have messages that indicate timestamp extraction failed

01-31-2024 15:08:17.539 +0300 WARN DateParserVerbose [17276 merging_0] - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (20) characters of event. Defaulting to timestamp of previous event (Wed Jan 31 15:08:05 2024). Context: source=udp:1100|host=172.22.0.11|wlc_syslog|\r\n 566 similar messages suppressed. First occurred at: Wed Jan 31 15:03:13 2024

All events from this sourcetype look like this:

<172>hostname: *spamApTask0: Jan 31 12:58:47.692: %LWAPP-4-SIG_INFO1: [PA]spam_lrad.c:56582 Signature information; AP 00:57:d2:86:c0:30, alarm ON, standard sig Auth flood, track per-Macprecedence 5, hits 300, slot 0, channel 1, most offending MAC 54:14:f3:c8:a1:b3

Before asking, I tried to find events without a timestamp by using regex and cluster commands but didn't find anything

So, is it normal behavior, and splunk indicates timestamp absence before moving to nullQueue or did I do something wrong?

Re: nullQueue and messages about problems with timestamp extraction

VatsalJagani — Wed, 31 Jan 2024 16:59:40 GMT

@NoSpaces - That could be true because Splunk applies Timestamp Parsing before Transforms (nullQueue).

I hope this helps to understand why you are seeing the log. Please upvote and accept if this helps and resolves your query!!!

Re: nullQueue and messages about problems with timestamp extraction

richgalloway — Wed, 31 Jan 2024 18:25:00 GMT

You won't find events without a timestamp because Splunk always stores every event with a timestamp. If the event does not come with a timestamp or if the timestamp is invalid then Splunk will use the timestamp from the previous event.

The timestamp warning cited does not apply to same sourcetype as the nullQueue transform. The warning is for wlc_syslog and the transform is for wlc_syslog_rt0.

Re: nullQueue and messages about problems with timestamp extraction

PickleRick — Wed, 31 Jan 2024 19:22:50 GMT

See the Masa diagrams - https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774

Timestamp extraction is one of the very first steps in event processing. So even if you later decide to drop (send to nullQueue) some events, that will be done way later in the pipeline.

Re: nullQueue and messages about problems with timestamp extraction

NoSpaces — Thu, 01 Feb 2024 06:46:57 GMT

I can find events without timestamps by using regex
It is not about the _time field but about the existence of "time" at the event
Apparanly, my first explanation is not good enough 😃

Re: nullQueue and messages about problems with timestamp extraction

NoSpaces — Thu, 01 Feb 2024 06:48:00 GMT

Thank you for the link to the diagram