Getting Data In

How to set the timestamp for events with TIME_PREFIX after sourcetype renaming?

alexislh
Explorer

Hi all,

I have this kind of log from 1 source :

DateLog=1459870479.000 ... TypeLog=Syslog ... Apr  5 17:34:37.618 ...
DateLog=1459870479.000 ... TypeLog=Trap ...

For some reason the DateLog timestamp is slightly off, and I would like to use the other one in Syslog messages instead: Apr 5 17:34:37.618

I succeed doing that by setting a TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD.

Problem is, as you can see, this timestamp does not appear in the Trap messages, so its timestamp is not recognized (no fallback or the like).

So I have tried to rename the source type in transforms & props:

TypeLog=Syslog > sourcetype = sourctetype:syslog
TypeLog=Trap > sourcetype = sourctetype:trap

Then I am trying to apply the TIME_PREFIX to this new sourcetype in props :

[sourctetype:syslog]
TIME_PREFIX = ...

But it does not work.

I don't remember the exact order in which Splunk indexes the events ,but I guess the timestamp is applied before the sourcetype is transformed, and then it's kinda too late for what I want to do.

Anyone gone through this before?

Thanks in advance for the ideas.

Alexis

0 Karma
1 Solution

neilrc
Explorer

Hi there,

You could approach this in a couple of ways. Maybe splitting out the data sources into 2 different files and then configure splunk for each could be the easiest bet. If this is not an option:

To fix sourcetype you could create a props.conf config on source and reference a transform that assigns the sourcetype:

props.conf:

[source:...]
TRANSFORMS-sourcetype = syslogSourcetype, trapSourcetype

transforms.conf

[syslogSourcetype]
REGEX= TypeLog=Syslog
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::syslog

[trapSourcetype]
REGEX= TypeLog=Trap
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::trap

As for the timestamp recognition you could look into a custom datetime.xml config.

http://blogs.splunk.com/2014/04/23/its-that-time-again/

Cheers

View solution in original post

neilrc
Explorer

Hi there,

You could approach this in a couple of ways. Maybe splitting out the data sources into 2 different files and then configure splunk for each could be the easiest bet. If this is not an option:

To fix sourcetype you could create a props.conf config on source and reference a transform that assigns the sourcetype:

props.conf:

[source:...]
TRANSFORMS-sourcetype = syslogSourcetype, trapSourcetype

transforms.conf

[syslogSourcetype]
REGEX= TypeLog=Syslog
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::syslog

[trapSourcetype]
REGEX= TypeLog=Trap
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::trap

As for the timestamp recognition you could look into a custom datetime.xml config.

http://blogs.splunk.com/2014/04/23/its-that-time-again/

Cheers

alexislh
Explorer

Hi neilrc,

Yeah, actually the source is obtained via db connect so yes indeed, could be more simple to have an action there by adjusting the SQL query and split the input in two with respective sourcetype set.

Thanks for the idea 🙂 !

Alexis

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...