topic Re: tcpin_cooked_pqueue blocking in Monitoring Splunk

tcpin_cooked_pqueue blocking

triest — Tue, 29 Sep 2020 23:01:18 GMT

I've recently made a career change, so I have a new Splunk environment where they leverage intermediary forwarders. Two of the intermediary forwarders are having their tcpin_cooked_pqueue fill which causes blocking. I would really appreciate some help troubleshooting and coming up with a suggested fix.

1, Since the tcpin_cooked queue is very early, the first question is obviously are later queues filling causing a backup; that's not the case only the tcpin cooked queue is filling. Also, parallel queues are enabled and set to 2.
2. Once the business day is over, the queue quickly empties.

3. The intermediary forwarders (where the queue filling happens) are physical systems running Suse Enteprise Server 11 with a load average around 2 during the day (1 processor, 16 cores, 32 threads), are using about 5.5GB of the available 32GB of memory. Network wise its receiving around 300KB/s and transmitting around 3005KB/s and has about 400 forwarders connected to it.
3. In terms of ulimits:
virtual address space size: unlimited
data segment size: unlimited
resident memory size: unlimited
stack size: 8388608 bytes [hard maximum: unlimited]
core file size: 1024 bytes [hard maximum: unlimited]
data file size: unlimited
open files: 10240 files
user processes: 256476 processes
cpu time: unlimited
Linux transparent hugepage support, enabled="never" defrag="never"
Linux vm.overcommit setting, value="0"

The key maybe that the forwarders sending typically are coming over fairly low bandwidth connections, so that may cause a lot of network connections per fairly low data ingestion rate.

Re: tcpin_cooked_pqueue blocking

skirven — Sun, 22 Nov 2020 17:38:52 GMT

Hi!

I ran across this when researching parallelism on Heavy Forwarders. Did you ever get a resolution here? I was curious if you increased your parallel value or not?
Thanks!

Stephen

Re: tcpin_cooked_pqueue blocking

isoutamo — Sun, 22 Nov 2020 22:00:25 GMT

Based on my experience, on physical machine it’s good to use parallel pipelines. Have you some bottleneck or why you are looking this?
Btw. You could add HFs as indexers on MC to better analyze what there is happening. On ideas.splunk.com there is a proposal to add HF as own role in MC, which you could vote if this is what you are needing.
r. Ismo

Re: tcpin_cooked_pqueue blocking

skirven — Mon, 23 Nov 2020 11:36:09 GMT

For my use case, I'm actually trying to facilitate better Search Peer data distribution. So if my Internediate HF (which is a VM. 😞 ) had 2 pipelines, would it not then accept 2 streams, and send to potentially 2 different indexers at the same time? So if I have 5 HFs, I could theoretically feed 10 Search Peers at the same time?

That may be slightly off topic here, so I may create a new topic. And I'll have to find the Idea for the HF on the DMC. That would be cool!

Stephen

Re: tcpin_cooked_pqueue blocking

isoutamo — Mon, 23 Nov 2020 11:55:15 GMT

Definitely it works better if you add second pipeline.
I think that this conf presentation will help you a lot: https://conf.splunk.com/files/2019/slides/FN1402.pdf

Re: tcpin_cooked_pqueue blocking

skirven — Mon, 23 Nov 2020 11:59:47 GMT

Thanks! I was at .conf last year, and totally didn't see this! I was dealing with other tech debt at the time. We've made a lot of progress since then. 🙂 I'll have to pull the talk and listen to it.

-Stephen