Monitoring Splunk

tcpin_cooked_pqueue blocking

triest
Communicator

I've recently made a career change, so I have a new Splunk environment where they leverage intermediary forwarders. Two of the intermediary forwarders are having their tcpin_cooked_pqueue fill which causes blocking. I would really appreciate some help troubleshooting and coming up with a suggested fix.

1, Since the tcpin_cooked queue is very early, the first question is obviously are later queues filling causing a backup; that's not the case only the tcpin cooked queue is filling. Also, parallel queues are enabled and set to 2.
2. Once the business day is over, the queue quickly empties.

3. The intermediary forwarders (where the queue filling happens) are physical systems running Suse Enteprise Server 11 with a load average around 2 during the day (1 processor, 16 cores, 32 threads), are using about 5.5GB of the available 32GB of memory. Network wise its receiving around 300KB/s and transmitting around 3005KB/s and has about 400 forwarders connected to it.
3. In terms of ulimits:
virtual address space size: unlimited
data segment size: unlimited
resident memory size: unlimited
stack size: 8388608 bytes [hard maximum: unlimited]
core file size: 1024 bytes [hard maximum: unlimited]
data file size: unlimited
open files: 10240 files
user processes: 256476 processes
cpu time: unlimited
Linux transparent hugepage support, enabled="never" defrag="never"
Linux vm.overcommit setting, value="0"

The key maybe that the forwarders sending typically are coming over fairly low bandwidth connections, so that may cause a lot of network connections per fairly low data ingestion rate.

Tags (1)
0 Karma

skirven
Communicator

Hi!

    I ran across this when researching parallelism on Heavy Forwarders. Did you ever get a resolution here? I was curious if you increased your parallel value or not? 
Thanks!

Stephen

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Based on my experience, on physical machine it’s good to use parallel pipelines. Have you some bottleneck or why you are looking this?
Btw. You could add HFs as indexers on MC to better analyze what there is happening. On ideas.splunk.com there is a proposal to add HF as own role in MC, which you could vote if this is what you are needing.
r. Ismo

skirven
Communicator

For my use case, I'm actually trying to facilitate better Search Peer data distribution. So if my Internediate HF (which is a VM. 😞 ) had 2 pipelines, would it not then accept 2 streams, and send to potentially 2 different indexers at the same time? So if I have 5 HFs, I could theoretically feed 10 Search Peers at the same time?

That may be slightly off topic here, so I may create a new topic. And I'll have to find the Idea for the HF on the DMC. That would be cool!

Stephen

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Definitely it works better if you add second pipeline.
I think that this conf presentation will help you a lot: https://conf.splunk.com/files/2019/slides/FN1402.pdf
0 Karma

skirven
Communicator

Thanks! I was at .conf last year, and totally didn't see this! I was dealing with other tech debt at the time. We've made a lot of progress since then. 🙂 I'll have to pull the talk and listen to it.

-Stephen

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...