Monitoring Splunk

tcpin_cooked_pqueue blocking

triest
Communicator

I've recently made a career change, so I have a new Splunk environment where they leverage intermediary forwarders. Two of the intermediary forwarders are having their tcpin_cooked_pqueue fill which causes blocking. I would really appreciate some help troubleshooting and coming up with a suggested fix.

1, Since the tcpin_cooked queue is very early, the first question is obviously are later queues filling causing a backup; that's not the case only the tcpin cooked queue is filling. Also, parallel queues are enabled and set to 2.
2. Once the business day is over, the queue quickly empties.

3. The intermediary forwarders (where the queue filling happens) are physical systems running Suse Enteprise Server 11 with a load average around 2 during the day (1 processor, 16 cores, 32 threads), are using about 5.5GB of the available 32GB of memory. Network wise its receiving around 300KB/s and transmitting around 3005KB/s and has about 400 forwarders connected to it.
3. In terms of ulimits:
virtual address space size: unlimited
data segment size: unlimited
resident memory size: unlimited
stack size: 8388608 bytes [hard maximum: unlimited]
core file size: 1024 bytes [hard maximum: unlimited]
data file size: unlimited
open files: 10240 files
user processes: 256476 processes
cpu time: unlimited
Linux transparent hugepage support, enabled="never" defrag="never"
Linux vm.overcommit setting, value="0"

The key maybe that the forwarders sending typically are coming over fairly low bandwidth connections, so that may cause a lot of network connections per fairly low data ingestion rate.

Tags (1)
0 Karma

skirven
Path Finder

Hi!

    I ran across this when researching parallelism on Heavy Forwarders. Did you ever get a resolution here? I was curious if you increased your parallel value or not? 
Thanks!

Stephen

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Based on my experience, on physical machine it’s good to use parallel pipelines. Have you some bottleneck or why you are looking this?
Btw. You could add HFs as indexers on MC to better analyze what there is happening. On ideas.splunk.com there is a proposal to add HF as own role in MC, which you could vote if this is what you are needing.
r. Ismo

skirven
Path Finder

For my use case, I'm actually trying to facilitate better Search Peer data distribution. So if my Internediate HF (which is a VM. 😞 ) had 2 pipelines, would it not then accept 2 streams, and send to potentially 2 different indexers at the same time? So if I have 5 HFs, I could theoretically feed 10 Search Peers at the same time?

That may be slightly off topic here, so I may create a new topic. And I'll have to find the Idea for the HF on the DMC. That would be cool!

Stephen

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Definitely it works better if you add second pipeline.
I think that this conf presentation will help you a lot: https://conf.splunk.com/files/2019/slides/FN1402.pdf
0 Karma

skirven
Path Finder

Thanks! I was at .conf last year, and totally didn't see this! I was dealing with other tech debt at the time. We've made a lot of progress since then. 🙂 I'll have to pull the talk and listen to it.

-Stephen

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...