More syslog forwarding fun...

[Steve_Litras]

So I've got forwarding of splunk data set up for certain systems in my environment to go to a 3rd party, in addition to through splunk. However, it looks like the host field doesn't get included in that forward, so the 3rd party just sees all this data coming from one system, with no differentiation of where the events originally came from (they're coming from a light forwarder, through an intermediate forwarding layer, into the indexing layer and being forwarded at the index layer via syslog).

Is the host field being dropped? Is there a way to add it back in?

[gfriedmann]

I've worked with multiple systems and here are two things to try individually or together 1) Make sure the log message outputed by the output will include the original origin IP address as the first part of the message... just after the priority and timestamp if it is present. in syslog-ng, this sort of formatting looks like "template("$STAMP $HOST $MSG\n")"

2) Try using TCP. Some receivers take TCP transport as a cue that the soruce might be a forwarder with some intelligence.

Regarding truncation before an expected 1.5k max, i have seen truncation at half the set value in some systems due to UTF16 representation doubling the amount of interpreted length somewhere along the way.

[gkanapathy]

Make sure you are not setting syslogSourceType in your output group configuration. You should probably also set priority and timestampformat. I believe that setting syslogSourceType to something other than "syslog" (or whatever the default is) stops Splunk from prepending the host.

You will also need to set timestampformat if you want Splunk to add the Splunk event timestamp, or you can try to count on the target server to prepend the current time.

[Steve_Litras]

I'm not using setting syslogSourceType anywhere in my config. I did some tcpdumps on the server, and it looks like all the syslog output it getting truncated, but not at a consistent length (well under the 1.5K MTU). I haven't changed the default truncation value.