Splunk Search

How to configure extraction for multiple timestamps from different message formats going to the same index?

_smp_
Builder

Hi.

I have two message formats falling into the same index. One of these message formats only contains one timestamp without milliseconds, but the second format contains a second timestamp with milliseconds. For reference:

Message format #1 with two timestamps, the second timestamp has milliseconds:

<181>Jan 27 15:15:26 monitor2 CISE_System_Statistics 0000006805 1 0 2017-01-27 15:15:26.036 +00:00 0000036144 70001 NOTICE System-Stats:

Message format #2 with one timestamp:

<180>Jan 27 15:15:32 CISE_Alarm WARN: RADIUS Authentication Request dropped :

I have configured the sourcetype to correctly pull out the second timestamp:

TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N %:z
TIME_PREFIX = \d{10}\s(?:\d\s){2}
SHOULD_LINEMERGE = true

This correctly processes format #1, but now events with message format #2 are merged into a single event. Is there a way to configure Splunk to pick the second timestamp from format #1, but the first timestamp for format #2?

0 Karma

woodcock
Esteemed Legend

You can do this using a custom datetime.xml file; read about it here:

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

0 Karma

coltwanger
Contributor

Perhaps you can configure the second event to route to a new sourcetype, where you can specify the second time format?

You could have:

[sourcetype:withMS]
 TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N %:z
 TIME_PREFIX = \d{10}\s(?:\d\s){2}

[sourcetype:withoutMS]
TIME_FORMAT = %b %d %H:%M:%S  #just a guess
TIME_PREFIX = \<\d\d\d\>

Of course you would have to find a pattern in your events to match on in order to route it to a different sourcetype.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Routeandfilterdatad

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi scottprigge,
I don't think that is possible to have two distinct formats for timestamp of one sourcetype, the only way is to use two differente sourcetypes with two different timestamp formats.

If milliseconds of the first type are important for you, you could use the common format for index both the events types and manage time with millisecond of the first type in a dedicated field (not timestamp).

Bye.
Giuseppe

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...