Forwarding data from Heavy forwarder to syslog server

[pavanae]

I setup syslog output forwarding per the Splunk docs, but am not seeing anything being sent out nor receiving it on the endpoint.

Here is what I have applied on the heavyforwarder outputs.conf

[tcpout]
 defaultGroup = indexer_group,forwarders_syslog
 useACK = true

 [tcpout:indexer_group]
 server = indexer_ip_address:indexer:port
 clientCert = xxxxxxxx
 maxQueueSize = 20MB
 sslPassword = xxxxxxxxx

 [tcpout:forwarders_syslog]
 server = syslog_ip:syslog_port
 clientCert = xxxxxxx
 maxQueueSize = 20MB
 sslPassword = xxxxxxxx
 blockOnCloning = false
 dropClonedEventsOnQueueFull = 10
 useACK = false

Note :-
The configuration for forwarding the data to syslog can be found under [tcpout:forwarders_syslog]

The following errors are found on splunkd.log when the heavy forwarder trying to forward the logs to syslog server

 WARN  TcpOutputProc - Cooked connection to ip=syslog_ip:syslog_port timed out
 ERROR TcpOutputFd - Connection to host=syslog_ip:syslog_port failed
 WARN  TcpOutputFd - Connect to syslog_ip:syslog_port  failed. Connection refused

Also I do not see any connection issues when I'm trying to trouble shoot as follows :-

In heavy forwarder :-
Tried to telnet to the syslog server from heavyforwarder with the specified port and see that it's got conected.

In receiving server

netstat -tnlp | grep rsyslog

Tried the above and see that the specified port in Heavy forwarder is listening in TCP

Not sure where and what else should I be checking to transfer the data whatever the heavyforwarder is currently transffering to Indexer also to a syslog server.

[nickhills · Accepted solution]

You need a [syslog:<target_group>] not a [tcpout:forwarders_syslog] group.

remove:
,forwarders_syslog from [tcpout]

add:

[syslog]
defaultGroup = forwarders_syslog

Change the last stanza to

[syslog:forwarders_syslog]
  server = syslog_ip:syslog_port
  #the below options are not supported
  #clientCert = xxxxxxx
  #maxQueueSize = 20MB
  #sslPassword = xxxxxxxx
  #blockOnCloning = false
  #dropClonedEventsOnQueueFull = 10
  #useACK = false 

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf#Syslog_output----

Once you have that you need to configure routing in props.conf and transforms.conf
See: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd

But quick example.

props.conf

[my sourcetype]
TRANSFORMS-my_sourcetype_syslog = send_to_syslog

transforms.conf

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = forwarders_syslog

[pavanae]

Thank you @nickhillscpl Is it ok to specify 2 default groups as mentioned above.

1 default group for tcpout and the other for syslog?

[nickhills]

yes, because they are defaults for tcp (splunk2splunk) or syslog
You are just configuring a default group for each type of output.

[pavanae]

Thanks again @nickhills. Is it mandatory to have the props and transforms. what happens if I don't have those props and tranforms for the send_to_syslog.

[nickhills]

Sorry, I was not very clear.
The props and transforms allows you to selectively send sourcetypes for routing - in case you didn't want to send everything to syslog you can use the routing config to be specific about which ones you do.

With a default set, everything will get routed - if you only wanted a subset, remove the default group settings, and use the props/transforms.

If you want everything you should not need the props/transforms

[pavanae]

Got it Thanks @nickhillscpl. Not sure what else needs to be verified I still do not see anything coming to my syslog server. Is there any way to troubleshoot the connection?

[MuS]

Hi pavanae,

Try sending some events using nc see this here https://superuser.com/questions/1229415/simple-way-to-generate-syslog-over-tcp

cheers, MuS