Why does the heavy forwarder forwarding to external syslog stops working after a few minutes?

[ChrisW-TX]

Using HF to forward all events to Indexer and external syslog. When using syslog with tcp all processing basically stopped as the queues filled up (and I've adjusted queue sizes already).  I haven't found much on Internet about this but did try UDP with the thought that is should be "send and forget" as far as the HF is concerned so it shouldn't slow data ingestion down but it still does.

I'm not using a props or transforms for the syslog as I want it to send all events.  After bringing the HF up, within a few minutes the queues fill up and everything grinds to a halt. If you look at it from the local MC, you can see there is no resource load on the server and you see a little ingestion occur about every few minutes are so.  The little data that gets to the indexer gets more timestamp skewed.  

I'm beating my head on that proverbial rock as this was working fine with tcp for a while and now it isn't working even using UDP.

Here is my syslog outputs.conf on the HF:

[syslog]
defaultGroup = forwarders_syslog
maxQueueSize = 10MB

[syslog:forwarders_syslog]
server = xx.xx.xx.xx:10514
type = udp
disabled = 0
priority = <34>
timestampformat = %b %e %H:%M:%S
useACK=false

I should also mention that there is no issue on the syslog server or the indexer, they are not taxed by any metric.  The syslog server is forwarding to another syslog via the Internet and does use tcp for that but since the incoming is written to a file, I don't see how that could impact the syslog receiving data from the HF.

Any advice will be appreciated.  I've opened a case with Splunk but they have been less than responsive.

[gcusello]

Hi @ChrisW-TX,

I had the same problem and I solved it in two ways:

initially I reduced the maxQueueSize, making some checks I found the correct dimension of this parameter that depends on many factors of your infrastructure (bandwidth, hardware resoulrse, data, etc...).

After I used the Splunk syslog connector (https://splunkbase.splunk.com/app/4740/) and I solved problems.

Ciao.

Giuseppe

[ChrisW-TX]

Thanks for the reply, however I don't think the Syslog Connector is applicable in my case as I'm not using Syslog to get data into Splunk.  I'm using Splunk to forward all incoming events (UF, HEC, etc.) to both an Indexer and a syslog relay that forwards to an external SIEM.  

I can try adjusting the maxQueueSize but I only modified that after this problem started.  

[gcusello]

Hi @ChrisW-TX,

you can use Splunk Syslog Connect App also to send logs to a third party.

Anyway the first solution should help you.

Ciao.

Giuseppe