Getting Data In

UDP:514 and source types

beano500
Engager

I wonder if someone could please explain to me how to achieve the following - I am running Splunk 4.2.4

I have a splunk index that I am using to capture logs from a number of Unix platform, I am doing this (for historic reasons) by using syslog to forward log events to the UDP port 514. I have this configured, and it works a treat.

But what I want is to be able to assign different source types to the incoming events (specifically I would like all events that look like sendmail to have the sourcetype=sendmail_syslog) so that I can then benefit from the inbuilt field recognition (specifically sendmail-qid).

Tags (2)
0 Karma
1 Solution

Starlette
Contributor

Sure that is possible and i do it all the time,,
Take a look at this, we call this sourcetype overriding

You should end up with something like this : ( replace syslog with your stanza for the syslog input)

props.conf

[syslog]
TRANSFORMS-sourcetype_and_host_override = sendmail
SHOULD_LINEMERGE = false

transforms.conf

[sendmail]
DEST_KEY = MetaData:Sourcetype
REGEX = (YOURREGEXTOCATCHSENDMAIL)
FORMAT = sourcetype::sendmail

Let me know if you succeed, otherwise paste a sample of your log to get the regex right...

View solution in original post

Starlette
Contributor

Sure that is possible and i do it all the time,,
Take a look at this, we call this sourcetype overriding

You should end up with something like this : ( replace syslog with your stanza for the syslog input)

props.conf

[syslog]
TRANSFORMS-sourcetype_and_host_override = sendmail
SHOULD_LINEMERGE = false

transforms.conf

[sendmail]
DEST_KEY = MetaData:Sourcetype
REGEX = (YOURREGEXTOCATCHSENDMAIL)
FORMAT = sourcetype::sendmail

Let me know if you succeed, otherwise paste a sample of your log to get the regex right...

View solution in original post

beano500
Engager

The 10% was the fact that the regex in the default/transforms.conf was not right for the log events I was receiving - so when I added a local (transforms.conf) stanza for [sendmail-extractions] with a REGEX and FORMAT that matched the actual log events being received, it all started to work.

0 Karma

beano500
Engager

OK - having done a lot of reading, and messing about - I now understand where I have been going wrong. Starlette - thanks for the original answer - it got me 90% of the way (I will explain the last 10% in the next comment). Though having read about field precedence, I do not understand why I need the 'SHOULD_LINEMERGE' in the props.conf/syslog stanza - and in my implementation I have left it out and it works fine.

0 Karma

Starlette
Contributor

The default is not picked anymore so you have to put them in local or under your app...
Arent those Field exactions not very high level btw?, and you want to add you own stuff as well?

So you better add those entries under you custom configs

system\local

props.conf

[sendmail]
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
TIME_FORMAT = %b %d %H:%M:%S
TRANSFORMS = syslog-host
REPORT-syslog = sendmail-extractions

transforms.conf
[sendmail-extractions]
REGEX = sendmail[(\d+)]: (\w+):
FORMAT = process::sendmail pid::$1 qid::$2

and your own extracts as well,,

0 Karma

beano500
Engager

Starlette - I made the configuration changed that you suggested, and newly indexed data has sourcetype=sendmail_syslog. As "sendmail_syslog" has mention in default/sourcetypes.conf and there are transforms (sendmail-pid and sendmail-qid) in default/transforms.conf - I was sort of hoping that splunk would then start pulling those fields.

0 Karma

Starlette
Contributor

But is the data after this config picked up (split) correctly?
If this was already sourcetyped data then you could use a sourcetype alias ( http://docs.splunk.com/Documentation/Splunk/latest/Data/Renamesourcetypes ), but thats not the case here.

I assume the data you mean already indexed is just a part of the syslogdata...the only way is to re-index is after clean your index. ( so from the source,,,syslog files maybee)
I played with indexed data forwarded as cloned data to a VM an reroute that back,,,but thats hard to explain here....

0 Karma

beano500
Engager

Thanks for this - I am able to set the sourcetype to be sendmail_syslog - but this raises a couple of other questions
(1) - this still does not appear to cause splunk to use the 'sendmail-qid' and 'sendmail-pid' defined in system/default/transforms.conf
(2) - as this change affects indexed data, is there an easy way to get splunk to re-index existing data to apply the changes retrospectively
Thanks

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!