Archive
Highlighted

Question about pipeline parallelization

Engager

How can I achieve pipeline parallelization in standalone Splunk indexer to optimize my CPU usage?
In Splunk 2016 .conf, it is mentioned to use above method if CPU is underutilized. For this, server.conf requires below changes:

parallelIngestionPipelines = 2

are there any other configuration changes required?

Also , do we need to configure in inputs.conf or in other configuration to bind these to use a specific pipeline processors or splunk takes care of this by own?

0 Karma
Highlighted

Re: Question about pipeline parallelization

Splunk Employee
Splunk Employee

This is the only configuration option required for enabling additional pipelines. Once enabled, restart Splunk and it will work!

View solution in original post

Highlighted

Re: Question about pipeline parallelization

Splunk Employee
Splunk Employee

Yup. Just to add: parallelIngestionPipelines are very much dependent on your disk write performance, so make sure you are not constrained on that before you enable it.
Indexing is generally not a very CPU-intensive process, so your mileage may vary.

0 Karma
Highlighted

Re: Question about pipeline parallelization

Engager

@ssievert sure thank you for your prompt reply.
I will try it .

My typing queues are getting filled(because of too many extraction rules) very often which is causing downstream queues to get filled too . When i checked my CPU, it is underutilized so I thought if I enable parallelization it will clear the backlogs in pipeline processor.

What should I Consider about disk write performance before enabling parallelization?

Agan thanks in advance

0 Karma
Highlighted

Re: Question about pipeline parallelization

Splunk Employee
Splunk Employee

If you don't normally see your indexing queues blocked, you are probably fine. Start with two parallelIngestion pipelines and monitor behavior.

0 Karma
Highlighted

Re: Question about pipeline parallelization

Splunk Employee
Splunk Employee

Hey @aab5272, If esix solved your problem, please don't forget to accept an answer! You can upvote posts as well. (Karma points will be awarded for either action.) Happy Splunking!

0 Karma