I have two heavy forwarders that are responsible for sending syslog events via TCP to a third-party syslog server.
[host::<fqdn>] TRANSFORMS-routing = send_to_syslog
[send_to_syslog] REGEX = . DEST_KEY = _SYSLOG_ROUTING FORMAT = syslog_siem
[syslog:syslog_siem] type = tcp server = <IP>:<port>
I am currently facing this error:
03-05-2017 00:41:43.058 +0800 ERROR DistributedClient - Write error The operation completed successfully. 03-05-2017 00:41:43.058 +0800 ERROR OutputProc - Failed to send data to <IP>:<port>. Failed to send data with TCPClient::send. err=-3
I am 100% sure it is not a network issue. The : is actually a load balancer IP address for the syslog server.
I have tried to use the same configuration to forward to a Splunk instance, and it works beautifully.
May I know what is wrong?
I would check firewalls are not blocking the traffic between Splunk and your syslog server. Can you also try using udp instead of tcp as a test?
splunk btool outputs list --debug
and confirm the outputs are being parsed correctly and there are no other config items overwritting your settings.
Besides forwarding the syslog events to a third-party syslog server, I am also forwarding certain syslog events to the indexer. I'm also using tcpout to send Windows event logs to port 9997 of the indexer.
[send_to_both] REGEX = . DEST_KEY = _SYSLOG_ROUTING FORMAT = syslog_siem, syslog_indexer [send_to_syslog] REGEX = . DEST_KEY = _SYSLOG_ROUTING FORMAT = syslog_siem
[syslog] defaultGroup = syslog_everything [syslog:syslog_siem] type = tcp server = <IP>:<port> [syslog:syslog_indexer] type = udp server = <IP>:<port> [syslog:syslog_everything] [tcpout] defaultGroup = send_to_indexer [tcpout:send_to_indexer] server = <IP>:9997 [tcpout-server://<IP>:9997]
It's not a firewall issue, as I am able to establish connection to the load balancer via the specified TCP port.
Unfortunately, it is impossible to change the connection to UDP.