Getting Data In
Highlighted

Changing sourcetype of incoming TCP syslogs

Hi everyone.

Quite new to the product, I am struggling a bit. All my logs are coming through syslog on TCP 514 and I am trying to get chance the sourcetype based on the name of the service logged, for example:

2010-11-03T16:54:26+01:00 canary.xxxxxxx.net named 17889 - - queries: info: client 127.0.0.1#57983: query: allsaintsfan-com-bk.mr.outblaze.com IN A +
2010-11-03T16:54:26+01:00 canary.xxxxxxx.net kernel - - - IPv6 addrconf: prefix with wrong length 56

Can anybody point me to the right direction? So far I have:

input.conf

[tcp://514]
connection_host = dns
source = syslog-tcp

props.conf

[source::syslog-tcp]
TRANSFORMS-sourcetype = override-sourcetype

transforms.conf

[override-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = \d{4}-?\d{2}-?\d{2}\D?\d{2}:\d{2}:\d{2}[+-]?\d{2}:\d{2}\s\S+\s(\S+)
FORMAT = sourcetype::$1

Can anybody figure out what's not working in there? Thank you in advance.

Highlighted

Re: Changing sourcetype of incoming TCP syslogs

Splunk Employee
Splunk Employee

A problem here (and the only problem I can see with a quick glance) is specifying SOURCE_KEY in transforms.conf. I'd leave that attribute blank so that your regex matches _raw, the holder of the raw text of the event. Other than that, you should be fine if your regex matches.

Highlighted

Re: Changing sourcetype of incoming TCP syslogs

Tried removing it but not go. I also removed the sourcetype=syslog from the inputs.conf but now sourcetype is just tcp_raw.

0 Karma
Highlighted

Re: Changing sourcetype of incoming TCP syslogs

Also tried a simple regex to make sure problem is not about matching... Still no go 😞

0 Karma
Highlighted

Re: Changing sourcetype of incoming TCP syslogs

Just to make sure I am not wrong on my test procedure, is restarting the splunkd enough to apply changes made to props.conf and transforms.conf?

Thanks.

0 Karma
Highlighted

Re: Changing sourcetype of incoming TCP syslogs

Explorer

Did you find an answer to your problem? I have a similar issue, which is why I ask.

As for your configuration, are you sure your syslog messages are arriving via TCP (as opposed to UDP)?

0 Karma
Highlighted

Re: Changing sourcetype of incoming TCP syslogs

Yes, my logs are coming in from TCP. I did not yet find an working answer. Hopefully you will 🙂

0 Karma
Highlighted

Re: Changing sourcetype of incoming TCP syslogs

Does it work on UDP?

0 Karma
Highlighted

Re: Changing sourcetype of incoming TCP syslogs

Explorer

Well I only mentioned UDP because that is what syslog uses by default (udp/514). I think you have to use syslog-ng to see syslog over TCP (tcp/514).

I'm still working on a resolution to this. Although I'm currently leaning towards this solution, where syslog messages are written to a file and then splunk parses the file:

http://answers.splunk.com/questions/8912/syslog-ng-filter-by-ip

0 Karma
Highlighted

Re: Changing sourcetype of incoming TCP syslogs

My syslog-ng is indeed configured to send logs as tcp flow. Parsing log files are fine, but seems to me that it is more configuration than necessary if the solution to this question could be found.
There is nothing vital in changing the sourcetype though, a simple process=xxx in a search would work as well when field is properly extracted.

0 Karma