I setup syslog output forwarding per the Splunk docs, but am not seeing anything being sent out nor receiving it on the endpoint.
All I'm trying to do is forward some data to syslog server via TCP port from a heavyforwarder. Here is what I have applied on the heavyforwarder outputs.conf
Outputs.conf on heavy forwarder
[syslog]
defaultGroup = forwarders_syslog
[syslog:forwarders_syslog]
server = syslog_hostname:port
clientCert = $SPLUNK_HOME/etc/auth/output-cert.pem
maxQueueSize = 20MB
sslPassword = xxxxxxx
type=tcp
sendCookedData=false
indexAndForward = 1
compressed = true
sslVerifyServerCert = false
Note :-
The configuration for forwarding the data to syslog can be found under [syslog:forwarders_syslog]
Props.conf on heavy forwarder
[sourcetype::XYZ]
TRANSFORMS-ABC_DEF= send_to_ABC_DEF
The following is transforms.conf on heavy forwarder
[send_to_ABC_DEF]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = forwarders_syslog
I tried the following troubleshooting steps to identify the root cause and don't find any
Able to telnet to the syslog server from heavy forwarder with the port specified in outputs.conf
tried the netstat -tnlp on the destination server and see the required port is listening and open.
seeing some traffic between source and destination.
Not sure what else should I be checking on to identify the root cause and fix the issue. Although I do see an error on splunkd.log as follows
ERROR OutputProc - Failed to send data to syslog_hostname:port. Failed to send data with TCPClient::send. err=-3
Also seeing the below blocked=true in metrics.log
INFO Metrics - group=queue, name=forwarders_syslog, blocked=true, max_size_kb=97, current_size_kb=97, current_size=147, largest_size=150, smallest_size=26
... View more