Getting Data In

Forwarding data from Heavy forwarder to syslog server

pavanae
Builder

I setup syslog output forwarding per the Splunk docs, but am not seeing anything being sent out nor receiving it on the endpoint.

Here is what I have applied on the heavyforwarder outputs.conf

[tcpout]
 defaultGroup = indexer_group,forwarders_syslog
 useACK = true

 [tcpout:indexer_group]
 server = indexer_ip_address:indexer:port
 clientCert = xxxxxxxx
 maxQueueSize = 20MB
 sslPassword = xxxxxxxxx

 [tcpout:forwarders_syslog]
 server = syslog_ip:syslog_port
 clientCert = xxxxxxx
 maxQueueSize = 20MB
 sslPassword = xxxxxxxx
 blockOnCloning = false
 dropClonedEventsOnQueueFull = 10
 useACK = false

Note :-
The configuration for forwarding the data to syslog can be found under [tcpout:forwarders_syslog]

The following errors are found on splunkd.log when the heavy forwarder trying to forward the logs to syslog server

 WARN  TcpOutputProc - Cooked connection to ip=syslog_ip:syslog_port timed out
 ERROR TcpOutputFd - Connection to host=syslog_ip:syslog_port failed
 WARN  TcpOutputFd - Connect to syslog_ip:syslog_port  failed. Connection refused

Also I do not see any connection issues when I'm trying to trouble shoot as follows :-

In heavy forwarder :-
Tried to telnet to the syslog server from heavyforwarder with the specified port and see that it's got conected.

In receiving server

netstat -tnlp | grep rsyslog

Tried the above and see that the specified port in Heavy forwarder is listening in TCP

Not sure where and what else should I be checking to transfer the data whatever the heavyforwarder is currently transffering to Indexer also to a syslog server.

0 Karma
1 Solution

nickhills
Ultra Champion

You need a [syslog:<target_group>] not a [tcpout:forwarders_syslog] group.

remove:
,forwarders_syslog from [tcpout]

add:

[syslog]
defaultGroup = forwarders_syslog

Change the last stanza to

[syslog:forwarders_syslog]
  server = syslog_ip:syslog_port
  #the below options are not supported
  #clientCert = xxxxxxx
  #maxQueueSize = 20MB
  #sslPassword = xxxxxxxx
  #blockOnCloning = false
  #dropClonedEventsOnQueueFull = 10
  #useACK = false 

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf#Syslog_output----

Once you have that you need to configure routing in props.conf and transforms.conf
See: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd

But quick example.

props.conf

[my sourcetype]
TRANSFORMS-my_sourcetype_syslog = send_to_syslog

transforms.conf

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = forwarders_syslog
If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

You need a [syslog:<target_group>] not a [tcpout:forwarders_syslog] group.

remove:
,forwarders_syslog from [tcpout]

add:

[syslog]
defaultGroup = forwarders_syslog

Change the last stanza to

[syslog:forwarders_syslog]
  server = syslog_ip:syslog_port
  #the below options are not supported
  #clientCert = xxxxxxx
  #maxQueueSize = 20MB
  #sslPassword = xxxxxxxx
  #blockOnCloning = false
  #dropClonedEventsOnQueueFull = 10
  #useACK = false 

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf#Syslog_output----

Once you have that you need to configure routing in props.conf and transforms.conf
See: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd

But quick example.

props.conf

[my sourcetype]
TRANSFORMS-my_sourcetype_syslog = send_to_syslog

transforms.conf

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = forwarders_syslog
If my comment helps, please give it a thumbs up!
0 Karma

pavanae
Builder

Thank you @nickhillscpl Is it ok to specify 2 default groups as mentioned above.

1 default group for tcpout and the other for syslog?

0 Karma

nickhills
Ultra Champion

yes, because they are defaults for tcp (splunk2splunk) or syslog
You are just configuring a default group for each type of output.

If my comment helps, please give it a thumbs up!
0 Karma

pavanae
Builder

Thanks again @nickhills. Is it mandatory to have the props and transforms. what happens if I don't have those props and tranforms for the send_to_syslog.

0 Karma

nickhills
Ultra Champion

Sorry, I was not very clear.
The props and transforms allows you to selectively send sourcetypes for routing - in case you didn't want to send everything to syslog you can use the routing config to be specific about which ones you do.

With a default set, everything will get routed - if you only wanted a subset, remove the default group settings, and use the props/transforms.

If you want everything you should not need the props/transforms

If my comment helps, please give it a thumbs up!
0 Karma

pavanae
Builder

Got it Thanks @nickhillscpl. Not sure what else needs to be verified I still do not see anything coming to my syslog server. Is there any way to troubleshoot the connection?

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi pavanae,

Try sending some events using nc see this here https://superuser.com/questions/1229415/simple-way-to-generate-syslog-over-tcp

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...