We're trying to forward data to a syslog server from a splunk server. However, seems that the hostname and process id tags are missing.
We've set up a central splunk server (10.10.40.9), and another splunk server to forward some data to a syslog server (10.10.40.10).
Configurations used: outputs.conf:
[tcpout] defaultGroup = 10.10.40.9_9997 [tcpout:10.10.40.9_9997] server = 10.10.40.9:9997 [tcpout-server://10.10.40.9:9997] [syslog:localhost_10514] server = 127.0.0.1:10514 type = tcp
[monitor:///var/log] disabled = false
[SYSLOG_FWD] REGEX = . DEST_KEY = _SYSLOG_ROUTING FORMAT = localhost_10514
example output messages (forwarded splunk output to syslog):
2010-10-29T17:13:05.557347+02:00 10-29-2010 17: 13:04.924 INFO Metrics - group=per_source_thruput, series="/opt/splunk/var/log/splunk/splunkd.log", kbps=2.094271, eps=20.466667, kb=62.828125 2010-10-29T17:13:05.557347+02:00 10-29-2010 17: 13:04.924 INFO Metrics - group=per_source_thruput, series="/opt/splunk/var/log/splunk/splunkd_stderr.log", kbps=0.000684, eps=0.033333, kb=0.020508
example desired output:
2010-10-23T08:29:08.087021+02:00 VCS02 glassfish-log: [#|2010-10-23T08:29:00.584+0200|INFO|glassfish3.0.1|null|_ThreadID=29;_ThreadName=Thread-1;|Total number of available updates : 0|#] 2010-10-24T01:38:48.570726+02:00 VCS02 ntpd: synchronized to 188.8.131.52, stratum 2 2010-10-24T04:02:08.609446+02:00 VCS02 rsyslogd: [origin software="rsyslogd" swVersion="5.6.0" x-pid="26020" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
The message should look like a standard syslog message (while I was testing I've used the same log that Splunk produces to send it to the syslog server):
$DATE $HOSTNAME $PROCESS[$PID] $MESSAGE
In order to receive RFC3164-compliant syslog output from Splunk (and namely, to include a time stamp in the sent event), you need to make sure to adequately set the
timestampformat configuration key in the
[syslog: stanza in outputs.conf, as per outputs.conf.spec:
timestampformat = <format> * If specified, the formatted timestamps are added to the start of events forwarded to syslog. * As above, this logic is only applied when the data is not syslog, or the syslogSourceType. * The format is a strftime-style timestamp formatting string. This is the same implementation used in the 'eval' search command, splunk logging, and other places in splunkd. * For example: %b %e %H:%M:%S * %b - Abbreviated month name (Jan, Feb, ...) * %e - Day of month * %H - Hour * %M - Minute * %s - Second * For a more exhaustive list of the formatting specifiers, refer to the online documentation. * Note that the string is not quoted. * Defaults to unset, which means that no timestamp will be inserted into the front of events.
server = syslog.splunk.com:514
type = tcp
timestampformat = %b %e %H:%M:%S
TRANSFORMS-routing = syslogrouting
REGEX = .
DESTKEY = SYSLOGROUTING
FORMAT = syslog_out
With this configuration, all events with sourcetype "syslog_test" will be routed to syslog host "syslog.splunk.com" on port TCP/514. The default priority code of "13" will be used, which is equivalent to "user.info", and a time stamp will be added as a prefix.
[root@beefysup01 43]# cat ../sources/test.log
This is a sample syslog event
[root@beefysup01 43]# $SPLUNKHOME/bin/splunk add oneshot ../sources/test.log -sourcetype syslogtest
Oneshot '/home/octavio/sources/test.log' added
[email@example.com:/]# nc -kl 514
<13>Jan 25 19:52:07 beefysup01 This is a sample syslog event
Note: I was unable to make this configuration work in versions earlier to Splunk 4.3. There are reports that this specific feature can fail in certain scenarios in 4.2.x. If you need this feature to work, please upgrade to Splunk 4.3.