How to set the timestamp for events with TIME_PREFIX after sourcetype renaming?

[alexislh]

Hi all,

I have this kind of log from 1 source :

DateLog=1459870479.000 ... TypeLog=Syslog ... Apr  5 17:34:37.618 ...
DateLog=1459870479.000 ... TypeLog=Trap ...

For some reason the DateLog timestamp is slightly off, and I would like to use the other one in Syslog messages instead: Apr 5 17:34:37.618

I succeed doing that by setting a TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD.

Problem is, as you can see, this timestamp does not appear in the Trap messages, so its timestamp is not recognized (no fallback or the like).

So I have tried to rename the source type in transforms & props:

TypeLog=Syslog > sourcetype = sourctetype:syslog
TypeLog=Trap > sourcetype = sourctetype:trap

Then I am trying to apply the TIME_PREFIX to this new sourcetype in props :

[sourctetype:syslog]
TIME_PREFIX = ...

But it does not work.

I don't remember the exact order in which Splunk indexes the events ,but I guess the timestamp is applied before the sourcetype is transformed, and then it's kinda too late for what I want to do.

Anyone gone through this before?

Thanks in advance for the ideas.

Alexis

[neilrc · Accepted solution]

Hi there,

You could approach this in a couple of ways. Maybe splitting out the data sources into 2 different files and then configure splunk for each could be the easiest bet. If this is not an option:

To fix sourcetype you could create a props.conf config on source and reference a transform that assigns the sourcetype:

props.conf:

[source:...]
TRANSFORMS-sourcetype = syslogSourcetype, trapSourcetype

transforms.conf

[syslogSourcetype]
REGEX= TypeLog=Syslog
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::syslog

[trapSourcetype]
REGEX= TypeLog=Trap
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::trap

As for the timestamp recognition you could look into a custom datetime.xml config.

http://blogs.splunk.com/2014/04/23/its-that-time-again/

Cheers

[alexislh]

Hi neilrc,

Yeah, actually the source is obtained via db connect so yes indeed, could be more simple to have an action there by adjusting the SQL query and split the input in two with respective sourcetype set.

Thanks for the idea 🙂 !

Alexis