Unable to forward syslog to third-party syslog server

[michaeltay]

I have two heavy forwarders that are responsible for sending syslog events via TCP to a third-party syslog server.

props.conf

[host::<fqdn>]
TRANSFORMS-routing = send_to_syslog

transforms.conf

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_siem

outputs.conf

[syslog:syslog_siem]
type = tcp
server = <IP>:<port>

I am currently facing this error:

03-05-2017 00:41:43.058 +0800 ERROR DistributedClient -  Write error The operation completed successfully.
03-05-2017 00:41:43.058 +0800 ERROR OutputProc - Failed to send data to <IP>:<port>. Failed to send data with TCPClient::send. err=-3

I am 100% sure it is not a network issue. The : is actually a load balancer IP address for the syslog server.

I have tried to use the same configuration to forward to a Splunk instance, and it works beautifully.

May I know what is wrong?

[sduff_splunk]

I would check firewalls are not blocking the traffic between Splunk and your syslog server. Can you also try using udp instead of tcp as a test?

Also run

 splunk btool outputs list --debug

and confirm the outputs are being parsed correctly and there are no other config items overwritting your settings.

[michaeltay]

Besides forwarding the syslog events to a third-party syslog server, I am also forwarding certain syslog events to the indexer. I'm also using tcpout to send Windows event logs to port 9997 of the indexer.

transforms.conf

[send_to_both]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_siem, syslog_indexer

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_siem

outputs.conf

[syslog]
defaultGroup = syslog_everything

[syslog:syslog_siem]
type = tcp
server = <IP>:<port>

[syslog:syslog_indexer]
type = udp
server = <IP>:<port>

[syslog:syslog_everything]

[tcpout]
defaultGroup = send_to_indexer

[tcpout:send_to_indexer]
server = <IP>:9997

[tcpout-server://<IP>:9997]

[michaeltay]

Hi sduff,

It's not a firewall issue, as I am able to establish connection to the load balancer via the specified TCP port.

Unfortunately, it is impossible to change the connection to UDP.