topic Re: How to Utilize all core on heavy forwarder on windows os in Deployment Architecture

How to Utilize all core on heavy forwarder on windows os

ips_mandar — Fri, 06 Dec 2019 05:29:31 GMT

I have one Heavy forwarder where zip files are ingested and parsed and hardware specification of HF is 64 GB Ram and 4-core, windows OS. I have set parallel ingestion pipeline to two. and my CPU utilization is around 60-65% and 2 cores are fully utilized while remaining two cores utilization is very less. so how could I tell splunk to utilize remaining two cores. is there any setting?
also indexing of data is very slow at indexer and after continuous indexing also 15GB data is getting indexed for whole day due to which backlog is increasing.
Please help me how can I utilize all 4 cores on HF?

Note- I have one Heavy forwarder and one indexer+search Head

Re: How to Utilize all core on heavy forwarder on windows os

gcusello — Fri, 06 Dec 2019 09:43:25 GMT

Hi @ips_mandar,
see with Splunk Monitor Console the real load on your HF and check if there are queues.
If you haven't big queues, my hint is to leave defaults.

At a first sight, I think that you don't need so much RAM on HF, but probably you have few CPUs, referring to Splunk HW reference ( https://docs.splunk.com/Documentation/Splunk/8.0.0/Capacity/Referencehardware ).

At the same time perform the same check on Indexer that you're sayng is very slow, probably you have to add more resources to it or add another one.
Anyway, a Splunk Indexers with the correct resources (at least 12 CPUs, 12 GB RAM) it's OK for until 100 GB/day.

But the main thing to check is storage: Splunk requires at least 800 IOPS, that means at least 8 SAS disks 15k on physical servers, on virtual servers you have to add more Indexers to parallelize indexing.
You can check IOPS using Bonnie++ open source tool.

Ciao.
Giuseppe

Re: How to Utilize all core on heavy forwarder on windows os

ips_mandar — Fri, 06 Dec 2019 11:09:16 GMT

Thanks @gcusello
My indexer hardware specification is very high (384 GB Physical Memory, 32 CPU Cores) so it won't be an issue at indexer level.
Yes there are queue block is happening at HF level and I did increased queue size of parsingqueue, aggqueue, aeq and typingqueue. still queue blockage is there but it's reduced than previously happening.
I have suggested my client to increase few cores but they are saying since now also it is not utilizing all 4 cores on HF so what will be use of adding few more cores?

Re: How to Utilize all core on heavy forwarder on windows os

gcusello — Fri, 06 Dec 2019 12:29:25 GMT

Hi @ips_mandar,
for Indexers check IOPS, especially if you have virtual servers!

Anyway, do on Indexer a check with Splunk Monitor Console because you can also have 32 CPUs but if they are all taken for heavy searches, indexing will be slow!
As example: one of my customers had 3 Search Heads with 16 CPUs each one but he had also a dashboard with 12 panels containing each one a real time search with 2 or 3 subsearches and this dashboard was used concurrently by 10-12 users: in Splunk each search (and subsearch) takes a CPU, so you can understand that the 48 available CPUs weren't sufficient for that load!

For HF, I don't think that the problem are Cores, at what level you have queue blocks?

If you open a ticket to Splunk Support (I always do it) they surely will highlight that the number of cores of HF are less the minimal specification, this could be a tool to speak with your customer.

Ciao.
Giuseppe