I setup syslog output forwarding per the Splunk docs, but am not seeing anything being sent out nor receiving it on the endpoint.
All I'm trying to do is forward some data to syslog server via TCP port from a heavyforwarder. Here is what I have applied on the heavyforwarder outputs.conf
Outputs.conf on heavy forwarder
[syslog]
defaultGroup = forwarders_syslog
[syslog:forwarders_syslog]
server = syslog_hostname:port
clientCert = $SPLUNK_HOME/etc/auth/output-cert.pem
maxQueueSize = 20MB
sslPassword = xxxxxxx
type=tcp
sendCookedData=false
indexAndForward = 1
compressed = true
sslVerifyServerCert = false
Note :-
The configuration for forwarding the data to syslog can be found under [syslog:forwarders_syslog]
Props.conf on heavy forwarder
[sourcetype::XYZ]
TRANSFORMS-ABC_DEF= send_to_ABC_DEF
The following is transforms.conf on heavy forwarder
[send_to_ABC_DEF]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = forwarders_syslog
I tried the following troubleshooting steps to identify the root cause and don't find any
Not sure what else should I be checking on to identify the root cause and fix the issue. Although I do see an error on splunkd.log as follows
ERROR OutputProc - Failed to send data to syslog_hostname:port. Failed to send data with TCPClient::send. err=-3
Also seeing the below blocked=true in metrics.log
INFO Metrics - group=queue, name=forwarders_syslog, blocked=true, max_size_kb=97, current_size_kb=97, current_size=147, largest_size=150, smallest_size=26
You have some minor mistakes in your configs.
Your props.conf is using the wrong stanza format.
Instead of [sourcetype::XYZ]
, it should be just [XYZ]
props.conf
[XYZ]
TRANSFORMS-ABC_DEF= send_to_ABC_DEF
sourcetype stanzas are just [sourcetypeName]
source stanzas use the format you originally used [source::yoursourcename]
host stanzas also use that format [host::yourhostname]
You also have some invalid options under [syslog:forwarders_syslog]
remove the following from outputs.conf as they have no effect for a syslog output:
clientCert = $SPLUNK_HOME/etc/auth/output-cert.pem
maxQueueSize = 20MB
sslPassword = xxxxxxx
sendCookedData=false
indexAndForward = 1
compressed = true
sslVerifyServerCert = false
Your remaining outputs.conf stanza should therefore just read:
[syslog:forwarders_syslog]
server = syslog_hostname:port
type=tcp
You have some minor mistakes in your configs.
Your props.conf is using the wrong stanza format.
Instead of [sourcetype::XYZ]
, it should be just [XYZ]
props.conf
[XYZ]
TRANSFORMS-ABC_DEF= send_to_ABC_DEF
sourcetype stanzas are just [sourcetypeName]
source stanzas use the format you originally used [source::yoursourcename]
host stanzas also use that format [host::yourhostname]
You also have some invalid options under [syslog:forwarders_syslog]
remove the following from outputs.conf as they have no effect for a syslog output:
clientCert = $SPLUNK_HOME/etc/auth/output-cert.pem
maxQueueSize = 20MB
sslPassword = xxxxxxx
sendCookedData=false
indexAndForward = 1
compressed = true
sslVerifyServerCert = false
Your remaining outputs.conf stanza should therefore just read:
[syslog:forwarders_syslog]
server = syslog_hostname:port
type=tcp
What is used in the syslog?Syslog-NG ,rsyslog ?? Can you paste the configuration of syslog?