topic Re: UDP:514 and source types in Getting Data In

UDP:514 and source types

beano500 — Sun, 13 Nov 2011 14:57:45 GMT

I wonder if someone could please explain to me how to achieve the following - I am running Splunk 4.2.4

I have a splunk index that I am using to capture logs from a number of Unix platform, I am doing this (for historic reasons) by using syslog to forward log events to the UDP port 514. I have this configured, and it works a treat.

But what I want is to be able to assign different source types to the incoming events (specifically I would like all events that look like sendmail to have the sourcetype=sendmail_syslog) so that I can then benefit from the inbuilt field recognition (specifically sendmail-qid).

Re: UDP:514 and source types

Starlette — Mon, 28 Sep 2020 10:05:50 GMT

Sure that is possible and i do it all the time,,
Take a look at this, we call this sourcetype overriding

You should end up with something like this : ( replace syslog with your stanza for the syslog input)

props.conf

[syslog]
TRANSFORMS-sourcetype_and_host_override = sendmail
SHOULD_LINEMERGE = false

transforms.conf

[sendmail]
DEST_KEY = MetaData:Sourcetype
REGEX = (YOURREGEXTOCATCHSENDMAIL)
FORMAT = sourcetype::sendmail

Let me know if you succeed, otherwise paste a sample of your log to get the regex right...

Re: UDP:514 and source types

beano500 — Sun, 13 Nov 2011 18:47:42 GMT

Thanks for this - I am able to set the sourcetype to be sendmail_syslog - but this raises a couple of other questions
(1) - this still does not appear to cause splunk to use the 'sendmail-qid' and 'sendmail-pid' defined in system/default/transforms.conf
(2) - as this change affects indexed data, is there an easy way to get splunk to re-index existing data to apply the changes retrospectively
Thanks

Re: UDP:514 and source types

Starlette — Sun, 13 Nov 2011 19:12:27 GMT

But is the data after this config picked up (split) correctly?
If this was already sourcetyped data then you could use a sourcetype alias ( http://docs.splunk.com/Documentation/Splunk/latest/Data/Renamesourcetypes ), but thats not the case here.

I assume the data you mean already indexed is just a part of the syslogdata...the only way is to re-index is after clean your index. ( so from the source,,,syslog files maybee)
I played with indexed data forwarded as cloned data to a VM an reroute that back,,,but thats hard to explain here....

Re: UDP:514 and source types

beano500 — Mon, 28 Sep 2020 10:06:08 GMT

Starlette - I made the configuration changed that you suggested, and newly indexed data has sourcetype=sendmail_syslog. As "sendmail_syslog" has mention in default/sourcetypes.conf and there are transforms (sendmail-pid and sendmail-qid) in default/transforms.conf - I was sort of hoping that splunk would then start pulling those fields.

Re: UDP:514 and source types

Starlette — Mon, 28 Sep 2020 10:06:11 GMT

The default is not picked anymore so you have to put them in local or under your app...
Arent those Field exactions not very high level btw?, and you want to add you own stuff as well?

So you better add those entries under you custom configs

system\local

props.conf

[sendmail]
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
TIME_FORMAT = %b %d %H:%M:%S
TRANSFORMS = syslog-host
REPORT-syslog = sendmail-extractions

transforms.conf
[sendmail-extractions]
REGEX = sendmail[(\d+)]: (\w+):
FORMAT = process::sendmail pid::$1 qid::$2

and your own extracts as well,,

Re: UDP:514 and source types

beano500 — Sat, 26 Nov 2011 21:33:45 GMT

OK - having done a lot of reading, and messing about - I now understand where I have been going wrong. Starlette - thanks for the original answer - it got me 90% of the way (I will explain the last 10% in the next comment). Though having read about field precedence, I do not understand why I need the 'SHOULD_LINEMERGE' in the props.conf/syslog stanza - and in my implementation I have left it out and it works fine.

Re: UDP:514 and source types

beano500 — Sat, 26 Nov 2011 21:33:57 GMT

The 10% was the fact that the regex in the default/transforms.conf was not right for the log events I was receiving - so when I added a local (transforms.conf) stanza for [sendmail-extractions] with a REGEX and FORMAT that matched the actual log events being received, it all started to work.