Having trouble to forward data to Kiwi syslog

[cemiam]

Hi,

I am trying forward data to Kiwi syslog. I have installed and configured a Heavy Forwarder and forward my syslog data to Heavy Forwarder. Then configured the HF to forward data to Kiwi syslog instance. However it is only forwarding cooked data and not forwarding the syslog data itself. I checked the network and it's all reachable. I have noticed below error. What should I do to overcome this issue?

Error:

Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 10 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Best Regards,

[mattymo · Accepted solution]

I believe there's 2 things you can try here,

1 : Use syslog output not TCP:
https://docs.splunk.com/Documentation/Splunk/7.0.0/Forwarding/Forwarddatatothird-partysystemsd#Syslo...

It looks like you have configured a TCP instead of a syslog output. I believe I would go this route as it uses a separate output processor, which should ensure you don't impact any output going to splunk...although looking at this config, I dont see one, so might not be a concern for this HF.

2: Update your TCP config to not send cooked data: https://docs.splunk.com/Documentation/Splunk/7.0.0/Forwarding/Forwarddatatothird-partysystemsd#TCP_d...

 [tcpout]

[tcpout:fastlane]
server = 10.1.1.35:6996
sendCookedData = false

While I hope this gets you rocking, I'd be questioning the design intent here, especially if we are catching udp from the input side on a port. If we are monitoring files...then I kind of get it....kind of.... 🙂

[cemiam]

Hi mmodestino,

Many thanks for the response. Soon I have started get logs. I have also getting audit logs. I have added sendCookedData = false parameter on [tcpout:fastlane] stanza. Do you have idea why is it still sending the audit logs?

Best Regards,

[mattymo]

yeah because of

/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf [tcpout]
.....
.....
     /opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_telemetry)

Those inputs are whitelisted by default. You can disable the inputs or just remove them from the whitelist.

I would also advise you try flipping over to syslog, as it wont have that settings...is there a reason you are remaining on tcp?

/opt/splunk/etc/system/default/outputs.conf               [syslog]
 /opt/splunk/etc/system/default/outputs.conf               maxEventSize = 1024
 /opt/splunk/etc/system/default/outputs.conf               priority = <13>
 /opt/splunk/etc/system/default/outputs.conf               type = udp

[cemiam]

I am still having trouble to disable audit logs. You can find the btool commands output below. Do you have any idea why am I still getting the audit logs?

/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf [syslog]
/opt/splunk/etc/system/local/outputs.conf defaultGroup = syslogG roup
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.0.white list = .*
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.1.black list = _.*
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.filter. disable = false
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf maxQueueSize = 500KB
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/local/outputs.conf [syslog:syslogGroup]
/opt/splunk/etc/system/local/outputs.conf sendCookedData = false
/opt/splunk/etc/system/local/outputs.conf server = 10.19.1.158:1 514
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 1 00
/opt/splunk/etc/system/default/outputs.conf cipherSuite = ECDHE-EC DSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256: ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:EC DHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA 256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH -ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/local/outputs.conf defaultGroup = default -autolb-group
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueu eFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v 1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.0.white list = .*
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.1.black list = _.*
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.filter. disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 3 0
/opt/splunk/etc/system/local/outputs.conf indexAndForward = 1
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndex er = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf maxQueueSize = 500KB
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = fal se
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunk/etc/system/local/outputs.conf [tcpout-server://10.19 .1.158:514]
/opt/splunk/etc/system/local/outputs.conf [tcpout:default-autolb -group]
/opt/splunk/etc/system/local/outputs.conf disabled = false
/opt/splunk/etc/system/local/outputs.conf sendCookedData = false
/opt/splunk/etc/system/local/outputs.conf server = 10.19.1.158:5 14

[mattymo]

Hi cemiam,

Can you please share your outputs.conf config?

btool is a great command to get to know ;). run this from the cli of the HF:

./splunk btool outputs list --debug

https://docs.splunk.com/Documentation/Splunk/7.0.0/Troubleshooting/Usebtooltotroubleshootconfigurati...

Also, I have to ask....why bother sending to an HF then kiwi?? I would probably look at either a) just adding kiwi as secondary syslog target on the devices, or 2) put kiwi in front of splunk and use a UF eat logs kiwi puts down on disk?

As you are seeing, blocking of one of your outputs on the HF can affect the other....

[cemiam]

Hi mmodestino,

It was requested for a specific purpose. I don't have enough detail but I think this should work fine with current configuration. You can find outputs.conf config below.

/opt/splunk/etc/system/default/outputs.conf               [syslog]
/opt/splunk/etc/system/default/outputs.conf               maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf               priority = <13>
/opt/splunk/etc/system/default/outputs.conf               type = udp
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf               ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf               autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf               autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf               blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf               blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf               cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunk/etc/system/default/outputs.conf               compressed = false
/opt/splunk/etc/system/default/outputs.conf               connectionTimeout = 20
/opt/splunk/etc/system/local/outputs.conf                 defaultGroup = default-autolb-group
/opt/splunk/etc/system/default/outputs.conf               disabled = false
/opt/splunk/etc/system/default/outputs.conf               dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf               dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf               ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf               forceTimebasedAutoLB = false
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_telemetry)
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf               heartbeatFrequency = 30
/opt/splunk/etc/system/local/outputs.conf                 indexAndForward = 1
/opt/splunk/etc/system/default/outputs.conf               maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf               maxFailuresPerInterval = 2
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf maxQueueSize = 500KB
/opt/splunk/etc/system/default/outputs.conf               readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf               secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf               sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf               sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf               sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf               tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf               useACK = false
/opt/splunk/etc/system/default/outputs.conf               writeTimeout = 300
/opt/splunk/etc/system/local/outputs.conf                 [tcpout-server://10.19.1.xxx:514]
/opt/splunk/etc/system/local/outputs.conf                 [tcpout:default-autolb-group]
/opt/splunk/etc/system/local/outputs.conf                 disabled = false
/opt/splunk/etc/system/local/outputs.conf                 server = 10.19.1.xxx:514

[ncrisler]

cemiam,

I am assuming that your setup goes syslog data>Kiwi syslog>Splunk. Is this correct? It look like the reason why you are getting this error is because the Splunk Heavy Forwarder cannot connect to the indexer. The typical setup for syslog would be to have Kiwi write this data to disk and have a Universal Forwarder pick up the readable file on disk and send it up to be indexed. Let me know either way.

Here are some articles that might be of some use if you haven't seen them already:

https://answers.splunk.com/answers/290158/how-do-i-send-data-from-kiwi-syslog-to-a-splunk-in.html
https://answers.splunk.com/answers/80134/what-is-the-easiest-way-to-get-data-from-a-kiwi-syslog-serv...

[cemiam]

Hi ncrisler,

Thanks for the response. Actually it is like Syslog data > Heavy Forwarder > Kiwi. I have also suspected about the connection problem but Heavy forwarder sends the cooked data. There might be a configuration issue. I am not sure if the Heavy Forwarder gets the syslog messages but it is listening port 9997 and I have checked the connection on the syslog source.