Getting Data In

Changing sourcetype of incoming TCP syslogs

Alexandre_Nizou
Explorer

Hi everyone.

Quite new to the product, I am struggling a bit. All my logs are coming through syslog on TCP 514 and I am trying to get chance the sourcetype based on the name of the service logged, for example:

2010-11-03T16:54:26+01:00 canary.xxxxxxx.net named 17889 - - queries: info: client 127.0.0.1#57983: query: allsaintsfan-com-bk.mr.outblaze.com IN A +
2010-11-03T16:54:26+01:00 canary.xxxxxxx.net kernel - - - IPv6 addrconf: prefix with wrong length 56

Can anybody point me to the right direction? So far I have:

input.conf

[tcp://514]
connection_host = dns
source = syslog-tcp

props.conf

[source::syslog-tcp]
TRANSFORMS-sourcetype = override-sourcetype

transforms.conf

[override-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = \d{4}-?\d{2}-?\d{2}\D?\d{2}:\d{2}:\d{2}[+-]?\d{2}:\d{2}\s\S+\s(\S+)
FORMAT = sourcetype::$1

Can anybody figure out what's not working in there? Thank you in advance.

melipla
Explorer

Did you find an answer to your problem? I have a similar issue, which is why I ask.

As for your configuration, are you sure your syslog messages are arriving via TCP (as opposed to UDP)?

0 Karma

Alexandre_Nizou
Explorer

My syslog-ng is indeed configured to send logs as tcp flow. Parsing log files are fine, but seems to me that it is more configuration than necessary if the solution to this question could be found.
There is nothing vital in changing the sourcetype though, a simple process=xxx in a search would work as well when field is properly extracted.

0 Karma

melipla
Explorer

Well I only mentioned UDP because that is what syslog uses by default (udp/514). I think you have to use syslog-ng to see syslog over TCP (tcp/514).

I'm still working on a resolution to this. Although I'm currently leaning towards this solution, where syslog messages are written to a file and then splunk parses the file:

http://answers.splunk.com/questions/8912/syslog-ng-filter-by-ip

0 Karma

Alexandre_Nizou
Explorer

Does it work on UDP?

0 Karma

Alexandre_Nizou
Explorer

Yes, my logs are coming in from TCP. I did not yet find an working answer. Hopefully you will 🙂

0 Karma

Alexandre_Nizou
Explorer

Just to make sure I am not wrong on my test procedure, is restarting the splunkd enough to apply changes made to props.conf and transforms.conf?

Thanks.

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

A problem here (and the only problem I can see with a quick glance) is specifying SOURCE_KEY in transforms.conf. I'd leave that attribute blank so that your regex matches _raw, the holder of the raw text of the event. Other than that, you should be fine if your regex matches.

Alexandre_Nizou
Explorer

Also tried a simple regex to make sure problem is not about matching... Still no go 😞

0 Karma

Alexandre_Nizou
Explorer

Tried removing it but not go. I also removed the sourcetype=syslog from the inputs.conf but now sourcetype is just tcp_raw.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...