I have a Splunk indexer (splunk-4.0.9-74233-linux-2.6-x8664.rpm) sending cooked data to a Splunk forwarder (activegroup = Forwarder, splunk-4.2.2-101277-linux-2.6-x86_64.rpm) and from there to a syslog server (it has to be that way, don't ask).
This works, except that by the time the data is uncooked and fed to syslog, it's missing the priority, time stamp and host. I don't care about the priority, but the time stamp and host are important...
[tcpout:sfwd] server = 192.168.1.120:9997 type = tcp sendCookedData = true
[splunktcp://:9997] disabled = false _SYSLOG_ROUTING = syslog_to_loghost
[syslog] defaultGroup = syslog_to_loghost [syslog:syslog_to_loghost] disabled = false server = 192.168.1.100:514 type = udp
On the syslog loghost I get the test messages I feed to the indexer as simply:
Where I'm assuming the 13 is the Splunk-applied priority. Where is the time stamp and hostname?
What obvious thing am I doing wrong?
This seems awfully familiar to a bug I filed a while ago, SPL-50764: "Syslog output not RFC3164 compliant (missing headers)". There are some differences in that my issue is with that Splunk isn't adding host and time in the cases where these were not supplied in the incoming message either. Still, the same bug is likely to apply to your case as well. For reference I include my bug report below.
When forwarding events from Splunk as syslog data to a syslogd, Splunk does not add headers like it should in order to be RFC3164 compliant, which goes against what is said on http://www.splunk.com/base/Documentation/latest/Admin/Forwarddatatothird-partysystems : "The forwarder sends RFC 3164 compliant events to a TCP/UDP-based server and port, making the payload of any non-compliant data RFC 3164 compliant. "
Example setup showing the issue:
Splunk instance with a raw TCP listener and a configuration to forward events received on this input as syslog data.
[tcp://4711] connection_host = ip sourcetype = syslogtest
[syslogtest] TRANSFORMS-sl = send_to_syslog
[send_to_syslog] REGEX = . DEST_KEY = _SYSLOG_ROUTING FORMAT = syslog_test
[syslog:syslog_test] server = 192.168.1.4:514
In order to be RFC3164 compliant, syslog messages must include at least priority, timestamp and hostname. However, when connecting from the host "myhost" to TCP port 4711 on a Splunk server configured as in the setup above and sending a raw string such as "my test string", Splunk sends "<13>my test string" as syslog data. It should look something like "<13>Nov 9 22:01:12 myhost my test string".
Sadly no, I haven't found a workaround. It was an issue I had in a specific scenario a limited period of time, so I haven't followed up on the bug. I haven't heard of any updates though, so my guess is it is still open and unresolved.
In order to receive RFC3164-compliant syslog output from Splunk, you need to make sure to adequately set the
timestampformat configuration key in the
[syslog: stanza in outputs.conf, as per outputs.conf.spec:
timestampformat = <format> * If specified, the formatted timestamps are added to the start of events forwarded to syslog. * As above, this logic is only applied when the data is not syslog, or the syslogSourceType. * The format is a strftime-style timestamp formatting string. This is the same implementation used in the 'eval' search command, splunk logging, and other places in splunkd. * For example: %b %e %H:%M:%S * %b - Abbreviated month name (Jan, Feb, ...) * %e - Day of month * %H - Hour * %M - Minute * %s - Second * For a more exhaustive list of the formatting specifiers, refer to the online documentation. * Note that the string is not quoted. * Defaults to unset, which means that no timestamp will be inserted into the front of events.
server = syslog.splunk.com:514
type = tcp
timestampformat = %b %e %H:%M:%S
TRANSFORMS-routing = syslogrouting
REGEX = .
DESTKEY = SYSLOGROUTING
FORMAT = syslog_out
With this configuration, all events with sourcetype "syslog_test" will be routed to syslog host "syslog.splunk.com" on port TCP/514. The default priority code of "13" will be used, which is equivalent to "user.info", and a time stamp will be added as a prefix.
[root@beefysup01 43]# cat ../sources/test.log
This is a sample syslog event
[root@beefysup01 43]# $SPLUNKHOME/bin/splunk add oneshot ../sources/test.log -sourcetype syslogtest
Oneshot '/home/octavio/sources/test.log' added
[firstname.lastname@example.org:/]# nc -kl 514
<13>Jan 25 19:52:07 beefysup01 This is a sample syslog event
Note: I was unable to make this configuration work in versions earlier to Splunk 4.3. There are reports that this specific feature can fail in certain scenarios in 4.2.x. If you need this feature to work, please upgrade to Splunk 4.3.