topic Re: No time or host in forwarded syslog messages in Getting Data In

No time or host in forwarded syslog messages

nisse — Mon, 28 Sep 2020 09:46:37 GMT

I have a Splunk indexer (splunk-4.0.9-74233-linux-2.6-x86_64.rpm) sending cooked data to a Splunk forwarder (active_group = Forwarder, splunk-4.2.2-101277-linux-2.6-x86_64.rpm) and from there to a syslog server (it has to be that way, don't ask).

This works, except that by the time the data is uncooked and fed to syslog, it's missing the priority, time stamp and host. I don't care about the priority, but the time stamp and host are important...

Indexer /opt/splunk/etc/system/local/outputs.conf:

[tcpout:sfwd]
server = 192.168.1.120:9997
type = tcp
sendCookedData = true

Forwarder /opt/splunk/etc/system/local/inputs.conf:

[splunktcp://:9997]
disabled = false
_SYSLOG_ROUTING = syslog_to_loghost

Forwarder /opt/splunk/etc/system/local/outputs.conf:

[syslog]
defaultGroup = syslog_to_loghost

[syslog:syslog_to_loghost]
disabled = false
server = 192.168.1.100:514
type = udp

On the syslog loghost I get the test messages I feed to the indexer as simply:

<13>payload

Where I'm assuming the 13 is the Splunk-applied priority. Where is the time stamp and hostname?

What obvious thing am I doing wrong?

Re: No time or host in forwarded syslog messages

Ayn — Wed, 03 Aug 2011 07:13:50 GMT

This seems awfully familiar to a bug I filed a while ago, SPL-50764: "Syslog output not RFC3164 compliant (missing headers)". There are some differences in that my issue is with that Splunk isn't adding host and time in the cases where these were not supplied in the incoming message either. Still, the same bug is likely to apply to your case as well. For reference I include my bug report below.

When forwarding events from Splunk as syslog data to a syslogd, Splunk does not add headers like it should in order to be RFC3164 compliant, which goes against what is said on http://www.splunk.com/base/Documentation/latest/Admin/Forwarddatatothird-partysystems : "The forwarder sends RFC 3164 compliant events to a TCP/UDP-based server and port, making the payload of any non-compliant data RFC 3164 compliant. "

Example setup showing the issue:
Splunk instance with a raw TCP listener and a configuration to forward events received on this input as syslog data.

inputs.conf:

[tcp://4711]
connection_host = ip
sourcetype = syslogtest

props.conf:

[syslogtest]
TRANSFORMS-sl = send_to_syslog

transforms.conf:

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_test

outputs.conf:

[syslog:syslog_test]
server = 192.168.1.4:514

In order to be RFC3164 compliant, syslog messages must include at least priority, timestamp and hostname. However, when connecting from the host "myhost" to TCP port 4711 on a Splunk server configured as in the setup above and sending a raw string such as "my test string", Splunk sends "<13>my test string" as syslog data. It should look something like "<13>Nov 9 22:01:12 myhost my test string".

Re: No time or host in forwarded syslog messages

nisse — Wed, 03 Aug 2011 19:34:05 GMT

That sure sounds like what I'm seeing. Have you found a work-around? What's the bug status?

Re: No time or host in forwarded syslog messages

nisse — Wed, 03 Aug 2011 20:02:10 GMT

Just found almost the same question from 2010: http://splunk-base.splunk.com/answers/8514/forward-data-to-a-syslog-server but no answer there either.

Re: No time or host in forwarded syslog messages

Ayn — Wed, 03 Aug 2011 20:03:01 GMT

Sadly no, I haven't found a workaround. It was an issue I had in a specific scenario a limited period of time, so I haven't followed up on the bug. I haven't heard of any updates though, so my guess is it is still open and unresolved.

Re: No time or host in forwarded syslog messages

hexx — Mon, 28 Sep 2020 10:21:47 GMT

In order to receive RFC3164-compliant syslog output from Splunk, you need to make sure to adequately set the timestampformat configuration key in the [syslog:] stanza in outputs.conf, as per outputs.conf.spec:

timestampformat = <format>
* If specified, the formatted timestamps are added to the start of events forwarded to syslog.
* As above, this logic is only applied when the data is not syslog, or the syslogSourceType.
* The format is a strftime-style timestamp formatting string. This is the same implementation used in the 'eval' search command, splunk logging, and other places in splunkd.
    *  For example: %b %e %H:%M:%S
    * %b - Abbreviated month name (Jan, Feb, ...)
    * %e - Day of month
    * %H - Hour
    * %M - Minute
    * %s - Second
* For a more exhaustive list of the formatting specifiers, refer to the online documentation.
* Note that the string is not quoted.
* Defaults to unset, which means that no timestamp will be inserted into the front of events.

Test configuration (tested with Splunk 4.3 only😞

outputs.conf:

[syslog:syslog_out] server = syslog.splunk.com:514 type = tcp timestampformat = %b %e %H:%M:%S

props.conf:

[syslog_test] TRANSFORMS-routing = syslog_routing

transforms.conf:

[syslog_routing] REGEX = . DEST_KEY = _SYSLOG_ROUTING FORMAT = syslog_out

With this configuration, all events with sourcetype "syslog_test" will be routed to syslog host "syslog.splunk.com" on port TCP/514. The default priority code of "13" will be used, which is equivalent to "user.info", and a time stamp will be added as a prefix.

Example:

On the server hosting the Splunk instance:

[root@beefysup01 43]# cat ../sources/test.log This is a sample syslog event [root@beefysup01 43]# $SPLUNK_HOME/bin/splunk add oneshot ../sources/test.log -sourcetype syslog_test Oneshot '/home/octavio/sources/test.log' added

On the server recipient of the syslog forwarded messages, using netcat to receive the events:

[root@syslog.splunk.com:/]# nc -kl 514 <13>Jan 25 19:52:07 beefysup01 This is a sample syslog event

Note: I was unable to make this configuration work in versions earlier to Splunk 4.3. There are reports that this specific feature can fail in certain scenarios in 4.2.x. If you need this feature to work, please upgrade to Splunk 4.3.