Changing sourcetype of incoming TCP syslogs

[Alexandre_Nizou]

Hi everyone.

Quite new to the product, I am struggling a bit. All my logs are coming through syslog on TCP 514 and I am trying to get chance the sourcetype based on the name of the service logged, for example:

2010-11-03T16:54:26+01:00 canary.xxxxxxx.net named 17889 - - queries: info: client 127.0.0.1#57983: query: allsaintsfan-com-bk.mr.outblaze.com IN A +
2010-11-03T16:54:26+01:00 canary.xxxxxxx.net kernel - - - IPv6 addrconf: prefix with wrong length 56

Can anybody point me to the right direction? So far I have:

input.conf

[tcp://514]
connection_host = dns
source = syslog-tcp

props.conf

[source::syslog-tcp]
TRANSFORMS-sourcetype = override-sourcetype

transforms.conf

[override-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = \d{4}-?\d{2}-?\d{2}\D?\d{2}:\d{2}:\d{2}[+-]?\d{2}:\d{2}\s\S+\s(\S+)
FORMAT = sourcetype::$1

Can anybody figure out what's not working in there? Thank you in advance.

[melipla]

Did you find an answer to your problem? I have a similar issue, which is why I ask.

As for your configuration, are you sure your syslog messages are arriving via TCP (as opposed to UDP)?

[Alexandre_Nizou]

My syslog-ng is indeed configured to send logs as tcp flow. Parsing log files are fine, but seems to me that it is more configuration than necessary if the solution to this question could be found.
There is nothing vital in changing the sourcetype though, a simple process=xxx in a search would work as well when field is properly extracted.

[melipla]

Well I only mentioned UDP because that is what syslog uses by default (udp/514). I think you have to use syslog-ng to see syslog over TCP (tcp/514).

I'm still working on a resolution to this. Although I'm currently leaning towards this solution, where syslog messages are written to a file and then splunk parses the file:

http://answers.splunk.com/questions/8912/syslog-ng-filter-by-ip

[Alexandre_Nizou]

Does it work on UDP?

[Alexandre_Nizou]

Yes, my logs are coming in from TCP. I did not yet find an working answer. Hopefully you will 🙂

[Alexandre_Nizou]

Just to make sure I am not wrong on my test procedure, is restarting the splunkd enough to apply changes made to props.conf and transforms.conf?

Thanks.

[Stephen_Sorkin]

A problem here (and the only problem I can see with a quick glance) is specifying SOURCE_KEY in transforms.conf. I'd leave that attribute blank so that your regex matches _raw, the holder of the raw text of the event. Other than that, you should be fine if your regex matches.

[Alexandre_Nizou]

Also tried a simple regex to make sure problem is not about matching... Still no go 😞

[Alexandre_Nizou]

Tried removing it but not go. I also removed the sourcetype=syslog from the inputs.conf but now sourcetype is just tcp_raw.