topic Changing sourcetype of incoming TCP syslogs in Getting Data In

Changing sourcetype of incoming TCP syslogs

Alexandre_Nizou — Wed, 03 Nov 2010 22:59:34 GMT

Hi everyone.

Quite new to the product, I am struggling a bit. All my logs are coming through syslog on TCP 514 and I am trying to get chance the sourcetype based on the name of the service logged, for example:

2010-11-03T16:54:26+01:00 canary.xxxxxxx.net named 17889 - - queries: info: client 127.0.0.1#57983: query: allsaintsfan-com-bk.mr.outblaze.com IN A +
2010-11-03T16:54:26+01:00 canary.xxxxxxx.net kernel - - - IPv6 addrconf: prefix with wrong length 56

Can anybody point me to the right direction? So far I have:

input.conf

[tcp://514]
connection_host = dns
source = syslog-tcp

props.conf

[source::syslog-tcp]
TRANSFORMS-sourcetype = override-sourcetype

transforms.conf

[override-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = \d{4}-?\d{2}-?\d{2}\D?\d{2}:\d{2}:\d{2}[+-]?\d{2}:\d{2}\s\S+\s(\S+)
FORMAT = sourcetype::$1

Can anybody figure out what's not working in there? Thank you in advance.

Re: Changing sourcetype of incoming TCP syslogs

Stephen_Sorkin — Wed, 03 Nov 2010 23:23:45 GMT

A problem here (and the only problem I can see with a quick glance) is specifying SOURCE_KEY in transforms.conf. I'd leave that attribute blank so that your regex matches _raw, the holder of the raw text of the event. Other than that, you should be fine if your regex matches.

Re: Changing sourcetype of incoming TCP syslogs

Alexandre_Nizou — Wed, 03 Nov 2010 23:44:40 GMT

Tried removing it but not go. I also removed the sourcetype=syslog from the inputs.conf but now sourcetype is just tcp_raw.

Re: Changing sourcetype of incoming TCP syslogs

Alexandre_Nizou — Thu, 04 Nov 2010 17:09:28 GMT

Also tried a simple regex to make sure problem is not about matching... Still no go 😞

Re: Changing sourcetype of incoming TCP syslogs

Alexandre_Nizou — Thu, 04 Nov 2010 17:11:14 GMT

Just to make sure I am not wrong on my test procedure, is restarting the splunkd enough to apply changes made to props.conf and transforms.conf?

Thanks.

Re: Changing sourcetype of incoming TCP syslogs

melipla — Thu, 11 Nov 2010 09:50:24 GMT

Did you find an answer to your problem? I have a similar issue, which is why I ask.

As for your configuration, are you sure your syslog messages are arriving via TCP (as opposed to UDP)?

Re: Changing sourcetype of incoming TCP syslogs

Alexandre_Nizou — Tue, 16 Nov 2010 03:02:12 GMT

Yes, my logs are coming in from TCP. I did not yet find an working answer. Hopefully you will 🙂

Re: Changing sourcetype of incoming TCP syslogs

Alexandre_Nizou — Tue, 16 Nov 2010 03:05:41 GMT

Does it work on UDP?

Re: Changing sourcetype of incoming TCP syslogs

melipla — Tue, 16 Nov 2010 05:25:15 GMT

Well I only mentioned UDP because that is what syslog uses by default (udp/514). I think you have to use syslog-ng to see syslog over TCP (tcp/514).

I'm still working on a resolution to this. Although I'm currently leaning towards this solution, where syslog messages are written to a file and then splunk parses the file:

http://answers.splunk.com/questions/8912/syslog-ng-filter-by-ip

Re: Changing sourcetype of incoming TCP syslogs

Alexandre_Nizou — Wed, 17 Nov 2010 14:39:59 GMT

My syslog-ng is indeed configured to send logs as tcp flow. Parsing log files are fine, but seems to me that it is more configuration than necessary if the solution to this question could be found.
There is nothing vital in changing the sourcetype though, a simple process=xxx in a search would work as well when field is properly extracted.