Intermediate forwarders - queues blocked - TCP in refused for uptream forwarder

[_olivier_]

Hi splunkers. 

I would like to understand a tricky point.

I'm using a distributed environment with 2 intermediate universal forwarders. They have to deal with 1.2 TB of data per day.

1 - Strangely, these UF have their parsing queues used (TOP 1 of the queues usage !) and these forwarders are UF !!!

2 - These UF have 4 pipeline. If one of these pipeline parsing queue is full, the entire UF refuse connection from upstream forwarders.

There queues size where increased to 1GB (input / parsing / output ...). But sometimes, this situation comes back.

Have you got any idea what could hapening ?

[livehybrid]

Hi @_olivier_ 

Increasing the queue size isnt a solution for this problem, all you are doing is introducing a risk that if that host fails that you will lose 1GB of buffered/queued date (per pipeline!), this should only be used for smoothing out short bursts of data.

As @isoutamo has mentioned - having so many pipelines might not help you here, I believe an input will only make use of a single pipeline (whereas a pipeline can be be applied to multiple inputs). 

What kind of data are you sending to your UFs? Im suprised that the parsing queue is filling, as I wouldnt expect the UF to do much parsing. There is a change that its struggling to get the data out to the indexers, what is the connectivity like between the UFs and the indexers?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[_olivier_]

Hi Livehybrid

This pipelines configurations came from a splunk PS.

I understand the risk to loose 1 Gb of data if this forwarder goes down ! Thank you !

About the datas, all data is coming from upstreams forwarder. This is raw data (Firewall, DNS ...) and structured data as json entries.

The connectivity between f<dr and indexer is VPN (350 Mbps throughput for each forwarder).

A last point, i missed to wrote in my first post, We are in "y" double output  because we are mooving from one platform to a new one and during 1 month, we have to send data over the two platforms.

[isoutamo]

When you have more than one target where you are sending then if any of those will blocked the traffic then all traffic will be blocked quite soon after that. Basically after blocked targets queue is full then all other targets will be blocked. This is default behavior of splunk. There are two options which you could change to change this behavior but it means that probability of lost some event will increase.

[livehybrid]

Hi @_olivier_ 

I certainly don't doubt my PS colleagues in their recommendations, as they will certainly have more information that I do about this particular set up, but wanted to make sure you knew about the queue etc.

If its helpful, there is a really good explanation of pipelines at https://community.splunk.com/t5/Getting-Data-In/How-many-pipelines-should-I-use-on-a-forwarder/m-p/4... which is worth a read. 

Interesting that you say about sending to two platforms, as it does sound like congestion outbound from the UF rather than an actual issue with the parsing part of the pipeline. It might be worth (if possible) monitoring the network egress to confirm its not hitting a ceiling, and also check if either of the two outputs are blocking (Check the _internal logs in both platforms for "TcpOutputProc" errors.

Another resource worth checking is https://conf.splunk.com/files/2019/slides/FN1570.pdf which might also help.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[_olivier_]

Thanks for the links, I gona read them and check logs for output errors.

[isoutamo]

You definitely should read what Harendra said!

[isoutamo]

Even you have defined several pipelines there are only one input pipeline for input processors. See https://docs.splunk.com/Documentation/Splunk/9.4.2/Indexer/Pipelinesets that could lead you a situation what you described.

[_olivier_]

Hi isoutamo, 

In the splunk doc, the is the description for in an out pipelines : 

"When you enable multiple pipeline sets on a forwarder, each pipeline handles both data input and output."

Are you sure that only one line pipeline is affectied to input processor ?

[isoutamo]

What I have read and understand based on many discussions is that even there is several pipelines those all share only one input part. I have gotten understanding that there is one input and pipelines start after that and it’s possible that this inputs will be blocked which also blocks other pipelines.

[_olivier_]

OK, this is the explanation of the connexions refused when one pipeline queue get blocked.

Thanks,

Now, I have to understand why i've got pipelines queues blocked.

[isoutamo]

Usually process is that start to look from right to left and find first blocked / queue which is full. Then look the next processor of right hand side. Usually issue is there.