Raw TCP forwarded events contain ForwarderInfo in Splunk 5.0.1

[joelshprentz]

After upgrading to Splunk 5.0.1 from 4.3.1, TCP streams of forwarded events began to include lines such as this:

ForwarderInfo build=143156 version=5.0.1 os=Linux arch=x86_64 hostname=splunksrv1 guid=E2DDF88B-2F84-465E-99DE-CA4AA5CD6E71 fwdType=full ssl=false lastIndexer=None

ForwarderInfo usually appears on the first line of a TCP stream, but occasionally appears later in the stream or not at all.

These ForwarderInfo lines confuse our third-party application that receives forwarded events via TCP. The ForwarderInfo lines have a different format than other events and contain not date/time data.

Our outputs.conf remains unchanged since Splunk 4.3.1. We still specify sendCookedData = false. Here is an excerpt from outputs.conf:

[tcpout]
indexAndForward = true
defaultGroup = nothing

[tcpout:nothing]
disabled = false
server = falsefoo.bar.com:9998
dropEventsOnQueueFull = 1

[tcpout:app-event-dest]
server = appsrvr1:6001,appsrvr2:6001
autoLBFrequency = 45
sendCookedData = false

How can we suppress the ForwarderInfo lines in Splunk 5.0.1?

[mgildenhorn_spl]

The ForwarderInfo lines are the heartbeats being sent. If you don't need them, you can disable the heartbeats by adding a heartbeatFrequency attribute and setting it to 0. See below:

[tcpout]
indexAndForward = true
defaultGroup = nothing
heartbeatFrequency = 0

[delink]

I downvoted this post because heartbeat messages are still being sent even with sendcookeddata = false and heartbeatfrequency = 0. this is on version 6.2.7. it'd be really nice if splunk fixed this.

[kellycocat]

We had a similar problem as joelshprentz, and the above suggestion didn't end up working. Here's a similar thread regarding this same issue:

http://answers.splunk.com/answers/125955/3rd-party-syslog-server-recieved-the-strange-messages-from-...

We had to use the syslog forwarding method to not also get the heartbeat messages (disclaimer: I was doing the forwarding from a Heavy Forwarder, so the syslog forward method works perfectly in that scenario):

outputs.conf
[syslog:target_group]
server = host:port
type = tcp

props.conf
[spec]
TRANSFORMS-syslog=unique_name

transforms.conf
[unique_name]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = target_group

The above is what worked for us. Good luck.