Getting Data In

forward data to a syslog server

acalvo
Explorer

We're trying to forward data to a syslog server from a splunk server. However, seems that the hostname and process id tags are missing.

We've set up a central splunk server (10.10.40.9), and another splunk server to forward some data to a syslog server (10.10.40.10).

Configurations used: outputs.conf:

[tcpout]
defaultGroup = 10.10.40.9_9997

[tcpout:10.10.40.9_9997]
server = 10.10.40.9:9997

[tcpout-server://10.10.40.9:9997]

[syslog:localhost_10514]
server = 127.0.0.1:10514
type = tcp

inputs.conf:

[monitor:///var/log]
disabled = false

props.conf:

[host::*]
TRANSFORMS-routing=SYSLOG_FWD

transforms.conf:

[SYSLOG_FWD]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = localhost_10514

example output messages (forwarded splunk output to syslog):

2010-10-29T17:13:05.557347+02:00 10-29-2010 17: 13:04.924 INFO  Metrics - group=per_source_thruput, series="/opt/splunk/var/log/splunk/splunkd.log", kbps=2.094271, eps=20.466667, kb=62.828125
2010-10-29T17:13:05.557347+02:00 10-29-2010 17: 13:04.924 INFO  Metrics - group=per_source_thruput, series="/opt/splunk/var/log/splunk/splunkd_stderr.log", kbps=0.000684, eps=0.033333, kb=0.020508

example desired output:

2010-10-23T08:29:08.087021+02:00 VCS02 glassfish-log: [#|2010-10-23T08:29:00.584+0200|INFO|glassfish3.0.1|null|_ThreadID=29;_ThreadName=Thread-1;|Total number of available updates : 0|#]
2010-10-24T01:38:48.570726+02:00 VCS02 ntpd[14162]: synchronized to 77.226.252.14, stratum 2
2010-10-24T04:02:08.609446+02:00 VCS02 rsyslogd: [origin software="rsyslogd" swVersion="5.6.0" x-pid="26020" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

hexx
Splunk Employee
Splunk Employee

In order to receive RFC3164-compliant syslog output from Splunk (and namely, to include a time stamp in the sent event), you need to make sure to adequately set the timestampformat configuration key in the [syslog:] stanza in outputs.conf, as per outputs.conf.spec:

timestampformat = <format>
* If specified, the formatted timestamps are added to the start of events forwarded to syslog.
* As above, this logic is only applied when the data is not syslog, or the syslogSourceType.
* The format is a strftime-style timestamp formatting string. This is the same implementation used in the 'eval' search command, splunk logging, and other places in splunkd.
    *  For example: %b %e %H:%M:%S
    * %b - Abbreviated month name (Jan, Feb, ...)
    * %e - Day of month
    * %H - Hour
    * %M - Minute
    * %s - Second
* For a more exhaustive list of the formatting specifiers, refer to the online documentation.
* Note that the string is not quoted.
* Defaults to unset, which means that no timestamp will be inserted into the front of events.

Test configuration (tested with Splunk 4.3 only😞

  • outputs.conf:

[syslog:syslog_out]
server = syslog.splunk.com:514
type = tcp
timestampformat = %b %e %H:%M:%S

  • props.conf:

[syslog_test]
TRANSFORMS-routing = syslog_routing

  • transforms.conf:

[syslog_routing]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_out

With this configuration, all events with sourcetype "syslog_test" will be routed to syslog host "syslog.splunk.com" on port TCP/514. The default priority code of "13" will be used, which is equivalent to "user.info", and a time stamp will be added as a prefix.

Example:

  • On the server hosting the Splunk instance:

[root@beefysup01 43]# cat ../sources/test.log
This is a sample syslog event
[root@beefysup01 43]# $SPLUNK_HOME/bin/splunk add oneshot ../sources/test.log -sourcetype syslog_test
Oneshot '/home/octavio/sources/test.log' added

  • On the server recipient of the syslog forwarded messages, using netcat to receive the events:

[root@syslog.splunk.com:/]# nc -kl 514
<13>Jan 25 19:52:07 beefysup01 This is a sample syslog event

Note: I was unable to make this configuration work in versions earlier to Splunk 4.3. There are reports that this specific feature can fail in certain scenarios in 4.2.x. If you need this feature to work, please upgrade to Splunk 4.3.

nisse
Explorer
0 Karma

calyope7
New Member

Hello acalvo,

Did you find a resolution to this issue?

Thanks, Chris

0 Karma

acalvo
Explorer

The message should look like a standard syslog message (while I was testing I've used the same log that Splunk produces to send it to the syslog server):

$DATE $HOSTNAME $PROCESS[$PID] $MESSAGE

0 Karma

thepocketwade
Path Finder

Have you considered putting the syslog server in front of the Splunk server?

0 Karma

ftk
Motivator

What do you want the output to look like?

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...