Splunk Enterprise

TIME_FORMAT UDP Input Fail

matthewparry
Path Finder

Hi,

I am trying to use a different timestamp located in the event data. This is a UDP input on a forwarder which I have first created a new sourcetype.

props

[source::udp:514]
TRANSFORMS-changesource = new_sourcetype

transforms

[new_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = mydata
FORMAT = sourcetype::newsourcetype

Then on my indexer splunk is automatically appending the date + host to my events coming from UDP, this can be stopped by using "no_appending_timestamp = true" under the input and this actually causes splunk to see the correct date from the raw data.

I believe you cannot apply timestamp changes to sourcetypes that have been generated in a transform?

I have tried the following on the indexer but with no luck:

props

[host::192.168.1.1]
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_FORMAT = %b %d %H:%M:%S.%6N

Example data coming in via syslog-ng

Jul 30 12:57:02.68787 host1 testdaemon: DEBUG Starting daemon

This gets indexed

Jul 30 12:57:02 192.168.1.1 Jul 30 12:57:02.687871 host1 testdaemon: DEBUG Starting daemon

Help would be appreciated as I have rattled my brain for hours now! 🙂

Tags (1)
0 Karma
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

On your heavy forwarder, you've made some mistakes.

For props.conf, I would add TIME_PREFIX with a regex to match the data that comes before the timestamp you want to extract

[host::192.168.1.1]
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_PREFIX = REGEX
TIME_FORMAT = %b %d %H:%M:%S.%6N

This transform is incorrect:

FORMAT = sourcetype::newsourcetype

What you should be doing looks like this:

[new_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = mydata
FORMAT = newsourcetype

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

On your heavy forwarder, you've made some mistakes.

For props.conf, I would add TIME_PREFIX with a regex to match the data that comes before the timestamp you want to extract

[host::192.168.1.1]
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_PREFIX = REGEX
TIME_FORMAT = %b %d %H:%M:%S.%6N

This transform is incorrect:

FORMAT = sourcetype::newsourcetype

What you should be doing looks like this:

[new_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = mydata
FORMAT = newsourcetype

matthewparry
Path Finder

I am actually using:

[host::192.168.1.1]
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_PREFIX = .+(?=\w{3} \d{2} \d{2}:\d{2}:\d{2}.\d{6})
TIME_FORMAT = %b %d %H:%M:%S.%6N

This has worked. I am unable to get it too work by substituting the [host::192.168.1.1] for a sourcetype but I believe this is due to the fact I am using a transform to generate a new sourcetype and the timestamps need to come before any transforms..

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

only for search time field extractions, which I don't think is what is intended here:

FORMAT for search-time extractions:
* The format of this field as used during search time extractions is as follows:
* FORMAT = ::( ::)*
* where:
* field-name = [|$]
* field-value = [|$]
* Search-time extraction examples:
* 1. FORMAT = first::$1 second::$2 third::other-value
* 2. FORMAT = $1::$2

0 Karma

sowings
Splunk Employee
Splunk Employee

This document suggests that the "sourcetype::" is in fact needed.

0 Karma

matthewparry
Path Finder

I am trying to use the time from syslog-ng "Jul 30 12:57:02.687871" as the event time.

0 Karma

Drainy
Champion

No. once the sourcetype has been changed then it will have passed the index time parsing.
What you want to do is apply the timestamp stripping from the syslog message first and then change the sourcetype, e.g;

[source::udp:514]
TRANSFORMS-stripts = strip_timestamp
TRANSFORMS-changesource = new_sourcetype

You could still do it on a per host basis though, there may be another way to wriggle it round but this is the way I've ended up doing it on my implementations

0 Karma

matthewparry
Path Finder

I have a heavy forwarder that receives the UDP syslog data which then forwards this to the indexer.
For some reason when trying to create the source type, this needs to be on the forwarder? and the timestamp changes need to be done on the indexer?

This means it will never execute in the correct order?

0 Karma

sowings
Splunk Employee
Splunk Employee

You're trying to use the syslog-ng stamp at the beginning of the line, or the device's native stamp starting after its IP address?

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...