topic Re: Problem reading syslog events in Getting Data In

Problem reading syslog events

mmather67 — Thu, 20 Oct 2011 14:04:34 GMT

My firewall is using syslog-ng to send logs to my log server over TCP on port 514. In Splunk>>Manager>>Data inputs>>TCP I have one entry, for port 514, which says source=tcp:514x and host=Firewall.

If I set Sourcetype=syslog, one particular log appears with host=2011 instead of host=Firewall.

If instead I set Sourcetype=syslog-ng, most of the time a few events get combined into one.

What should I do?

Re: Problem reading syslog events

JSapienza — Thu, 20 Oct 2011 15:10:10 GMT

You might have a line format or line breaking issue. Are these multi-line events ? Paste in a few lines from the raw sylog so we can take a look.
What does the Stanza look like in your inputs.conf ? Check %SPLUNK_HOME%\etc\system\local\inputs.conf .

Re: Problem reading syslog events

mmather67 — Thu, 20 Oct 2011 17:36:27 GMT

In response to JSapienza

Syslog only provides single-line events. All examples below are single lines.

inputs.conf has nothing relevant.

When the sourcetype is syslog, this event is picked up properly:-

<190>2011:10:19-16:45:13 reverseproxy: srcip="211.142.x.x" localip="66.207.x.x" size="0" user="-" host="211.142.x.x" method="HEAD" statuscode="200" time="8772" url="/" server="66.207.x.x" referer="-" cookie="-" set-cookie="-"

and this one gets host=2011:-

<190>2011:10:19-16:45:13 reverseproxy: [Wed Oct 19 16:45:13 2011] [warn] [client 211.142.x.x] proxy: no HTTP 0.9 request (with no host line) on incoming request and preserve host set forcing hostname to be 66.207.x.x for uri /

When the sourcetype is syslog-ng, the following two events get picked up as one:-

<30>2011:10:20-06:49:13 ulogd[4729]: id="2002" severity="info" sys="SecureNet" sub="packetfilter" name="Packet accepted" action="accept" fwrule="17" initf="eth1" outitf="eth2" srcmac="0:1e:79:1a:x.x" dstmac="0:1a:8c:11:x.x" srcip="69.165.x.x" dstip="192.168.x.x" proto="6" length="60" tos="0x00" prec="0x00" ttl="56" srcport="60634" dstport="8000" tcpflags="SYN"

<30>2011:10:20-06:49:14 ulogd[4729]: id="2002" severity="info" sys="SecureNet" sub="packetfilter" name="Packet accepted" action="accept" fwrule="10" initf="eth0" outitf="eth2" srcmac="0:21:9b:8e:x.x" dstmac="0:1a:8c:11:x.x" srcip="192.168.x.x" dstip="192.168.x.x" proto="6" length="48" tos="0x00" prec="0x00" ttl="127" srcport="63563" dstport="9997" tcpflags="SYN"

By the way, the local props.conf says:

[source::tcp:514]

TIME_FORMAT = %Y:%m:%d-%H:%M:S

host=Firewall-props

but I don't believe that is relevant.

Re: Problem reading syslog events

JSapienza — Thu, 20 Oct 2011 18:28:08 GMT

You might try adding the fllowing stanza to %SPLUNK_HOME\etc\system\local\props.conf

[syslog-ng]
SHOULD_LINEMERGE = False

Bounce splunk and check your events.

Re: Problem reading syslog events

mmather67 — Thu, 20 Oct 2011 19:59:58 GMT

Excellent. Thanks for your help.

With the proviso that I don't know how to trigger host=2011, so I will wait for one of those events to happen naturally and see what happens.

...local\props.conf now says:

[syslog-ng]

TIME_FORMAT = %Y:%m:%d-%H:%M:%S

SHOULD_LINEMERGE=false

Is there anything else that should be done when changing the sourcetype from syslog to syslog-ng?

I presume, by the way, that the TCP 514 entry in Data Inputs applies before props.conf. Otherwise [syslog-ng] would not be recognised.

Re: Problem reading syslog events

Ayn — Thu, 20 Oct 2011 20:11:09 GMT

The reason you're getting host=2011 when using the "syslog" sourcetype is because Splunk has transforms for that particular sourcetype that sets the host based on log events. Here's the transform that does the job:

[syslog-host]
DEST_KEY = MetaData:Host
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
FORMAT = host::$1

Re: Problem reading syslog events

mmather67 — Thu, 20 Oct 2011 20:34:43 GMT

I cannot pretend to read that. But why is it doing it anyway? What is it hoping to achieve?