Getting Data In

Can we increase parallelIngestionPipelines in a heavy Forwarder?

koshyk
Super Champion

folks,
Have anyone tried configuring parallelIngestionPipelines on Heavy Forwarder? We have plenty of room for cpu/memory on heavy forwarder. Hence wanted to check if parallelIngestionPipelines can be increased in a Heavy Forwarder to forward to index cluster.

I can see documentation for Universal Forwarder & Indexer, but not on heavy forwarder.

0 Karma
1 Solution

ansif
Motivator

Yes you can. Refer the document below on Index Parallelization:

http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/Pipelinesets

How forwarders use multiple pipeline sets

When you enable multiple pipeline sets on a forwarder, each pipeline handles both data input and output. In the case of a heavy forwarder, each pipeline also handles parsing.

And we have this setting enabled as 2 and working as expected for a long.

View solution in original post

vanvan
Path Finder

Just beware - if your HF collect only a single input type, then the parallel ingestion won't work. E.g. TCP or UDP traffic for Syslog ingestion is not parallelized.

HansWurscht
Path Finder

Can you explain this in more detail?

Currently we are facing an issue, where we input http and syslog on our hf and mostly only one of our eight pipelines is is used.

0 Karma

vanvan
Path Finder

Yes,

What you probably do is what we did in the past - routing all syslog to our HFs on as single port, e.g. 514.
Splunk serves each port with single pipeline, so if you only use one port for all your syslog, then you will use only a sinigle pipeline. Better option is to configure different ports for different syslog sources (e.g. 1514 for firewalls, 2514 for NLBs, 3515 for WAFs, etc...) BUT the BEST option is to direct all syslog traffic into Syslog-NG service and write the data into files. Then use Universal Forwarder to monitor these files. This is by far the best solution for Syslog data.

itrimble1
Path Finder

We just enabled this on one of our Windows Event Collectors. Will take a few days to monitor performance.

0 Karma

ansif
Motivator

Yes you can. Refer the document below on Index Parallelization:

http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/Pipelinesets

How forwarders use multiple pipeline sets

When you enable multiple pipeline sets on a forwarder, each pipeline handles both data input and output. In the case of a heavy forwarder, each pipeline also handles parsing.

And we have this setting enabled as 2 and working as expected for a long.

dm1
Contributor

is there some metric or search we can use to verify if increasing the pipeline set actually has made some difference ?

0 Karma

koshyk
Super Champion

thanks for the confirmation @ansif

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...