syslog server not receiving any events from a heavyforwarder. what could be the issue and how can it be resolved?

[pavanae]

I setup syslog output forwarding per the Splunk docs, but am not seeing anything being sent out nor receiving it on the endpoint.

All I'm trying to do is forward some data to syslog server via TCP port from a heavyforwarder. Here is what I have applied on the heavyforwarder outputs.conf

Outputs.conf on heavy forwarder  

[syslog]
defaultGroup = forwarders_syslog

[syslog:forwarders_syslog]
server = syslog_hostname:port
clientCert = $SPLUNK_HOME/etc/auth/output-cert.pem
maxQueueSize = 20MB
sslPassword = xxxxxxx
type=tcp
sendCookedData=false
indexAndForward = 1
compressed = true
sslVerifyServerCert = false

Note :-
The configuration for forwarding the data to syslog can be found under [syslog:forwarders_syslog]

Props.conf on heavy forwarder

[sourcetype::XYZ]
TRANSFORMS-ABC_DEF= send_to_ABC_DEF

The following is transforms.conf on heavy forwarder

[send_to_ABC_DEF]
 REGEX = .
 DEST_KEY = _SYSLOG_ROUTING
 FORMAT = forwarders_syslog

I tried the following troubleshooting steps to identify the root cause and don't find any

1. Able to telnet to the syslog server from heavy forwarder with the port specified in outputs.conf
2. tried the netstat -tnlp on the destination server and see the required port is listening and open.
3. seeing some traffic between source and destination.

Not sure what else should I be checking on to identify the root cause and fix the issue. Although I do see an error on splunkd.log as follows

 ERROR OutputProc - Failed to send data to syslog_hostname:port. Failed to send data with TCPClient::send. err=-3

Also seeing the below blocked=true in metrics.log

INFO  Metrics - group=queue, name=forwarders_syslog, blocked=true, max_size_kb=97, current_size_kb=97, current_size=147, largest_size=150, smallest_size=26

[nickhills · Accepted solution]

You have some minor mistakes in your configs.

Your props.conf is using the wrong stanza format.
Instead of [sourcetype::XYZ], it should be just [XYZ]

props.conf

[XYZ]
TRANSFORMS-ABC_DEF= send_to_ABC_DEF

sourcetype stanzas are just [sourcetypeName]
source stanzas use the format you originally used [source::yoursourcename]
host stanzas also use that format [host::yourhostname]

You also have some invalid options under [syslog:forwarders_syslog]

remove the following from outputs.conf as they have no effect for a syslog output:

 clientCert = $SPLUNK_HOME/etc/auth/output-cert.pem
 maxQueueSize = 20MB
 sslPassword = xxxxxxx
 sendCookedData=false
 indexAndForward = 1
 compressed = true
 sslVerifyServerCert = false

Your remaining outputs.conf stanza should therefore just read:

[syslog:forwarders_syslog]
server = syslog_hostname:port
type=tcp

[ansif]

What is used in the syslog?Syslog-NG ,rsyslog ?? Can you paste the configuration of syslog?