This seems awfully familiar to a bug I filed a while ago, SPL-50764: "Syslog output not RFC3164 compliant (missing headers)". There are some differences in that my issue is with that Splunk isn't adding host and time in the cases where these were not supplied in the incoming message either. Still, the same bug is likely to apply to your case as well. For reference I include my bug report below.
When forwarding events from Splunk as syslog data to a syslogd, Splunk does not add headers like it should in order to be RFC3164 compliant, which goes against what is said on http://www.splunk.com/base/Documentation/latest/Admin/Forwarddatatothird-partysystems : "The forwarder sends RFC 3164 compliant events to a TCP/UDP-based server and port, making the payload of any non-compliant data RFC 3164 compliant. "
Example setup showing the issue:
Splunk instance with a raw TCP listener and a configuration to forward events received on this input as syslog data.
inputs.conf:
[tcp://4711]
connection_host = ip
sourcetype = syslogtest
props.conf:
[syslogtest]
TRANSFORMS-sl = send_to_syslog
transforms.conf:
[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_test
outputs.conf:
[syslog:syslog_test]
server = 192.168.1.4:514
In order to be RFC3164 compliant, syslog messages must include at least priority, timestamp and hostname. However, when connecting from the host "myhost" to TCP port 4711 on a Splunk server configured as in the setup above and sending a raw string such as "my test string", Splunk sends "<13>my test string" as syslog data. It should look something like "<13>Nov 9 22:01:12 myhost my test string".
... View more