Question about pipeline parallelization

[aab5272]

How can I achieve pipeline parallelization in standalone Splunk indexer to optimize my CPU usage?
In Splunk 2016 .conf, it is mentioned to use above method if CPU is underutilized. For this, server.conf requires below changes:

parallelIngestionPipelines = 2

are there any other configuration changes required?

Also , do we need to configure in inputs.conf or in other configuration to bind these to use a specific pipeline processors or splunk takes care of this by own?

[lfedak_splunk]

Hey @aab5272, If esix solved your problem, please don't forget to accept an answer! You can upvote posts as well. (Karma points will be awarded for either action.) Happy Splunking!

[esix_splunk · Accepted solution]

This is the only configuration option required for enabling additional pipelines. Once enabled, restart Splunk and it will work!

[s2_splunk]

Yup. Just to add: parallelIngestionPipelines are very much dependent on your disk write performance, so make sure you are not constrained on that before you enable it.
Indexing is generally not a very CPU-intensive process, so your mileage may vary.

[aab5272]

@ssievert sure thank you for your prompt reply.
I will try it .

My typing queues are getting filled(because of too many extraction rules) very often which is causing downstream queues to get filled too . When i checked my CPU, it is underutilized so I thought if I enable parallelization it will clear the backlogs in pipeline processor.

What should I Consider about disk write performance before enabling parallelization?

Agan thanks in advance

[s2_splunk]

If you don't normally see your indexing queues blocked, you are probably fine. Start with two parallelIngestion pipelines and monitor behavior.