topic Re: Question about pipeline parallelization in Splunk Enterprise

Question about pipeline parallelization

aab5272 — Wed, 20 Sep 2017 17:45:36 GMT

How can I achieve pipeline parallelization in standalone Splunk indexer to optimize my CPU usage?
In Splunk 2016 .conf, it is mentioned to use above method if CPU is underutilized. For this, server.conf requires below changes:

parallelIngestionPipelines = 2

are there any other configuration changes required?

Also , do we need to configure in inputs.conf or in other configuration to bind these to use a specific pipeline processors or splunk takes care of this by own?

Re: Question about pipeline parallelization

esix_splunk — Wed, 20 Sep 2017 17:53:55 GMT

This is the only configuration option required for enabling additional pipelines. Once enabled, restart Splunk and it will work!

Re: Question about pipeline parallelization

s2_splunk — Wed, 20 Sep 2017 17:59:10 GMT

Yup. Just to add: parallelIngestionPipelines are very much dependent on your disk write performance, so make sure you are not constrained on that before you enable it.
Indexing is generally not a very CPU-intensive process, so your mileage may vary.

Re: Question about pipeline parallelization

aab5272 — Wed, 20 Sep 2017 20:46:13 GMT

@ssievert sure thank you for your prompt reply.
I will try it .

My typing queues are getting filled(because of too many extraction rules) very often which is causing downstream queues to get filled too . When i checked my CPU, it is underutilized so I thought if I enable parallelization it will clear the backlogs in pipeline processor.

What should I Consider about disk write performance before enabling parallelization?

Agan thanks in advance

Re: Question about pipeline parallelization

lfedak_splunk — Thu, 21 Sep 2017 23:04:57 GMT

Hey @aab5272, If esix solved your problem, please don't forget to accept an answer! You can upvote posts as well. (Karma points will be awarded for either action.) Happy Splunking!

Re: Question about pipeline parallelization

s2_splunk — Fri, 22 Sep 2017 01:53:15 GMT

If you don't normally see your indexing queues blocked, you are probably fine. Start with two parallelIngestion pipelines and monitor behavior.