Getting Data In

What is multithreaded ingestion?

New Member

Does anyone know what is meant by the term "multithreaded ingestion"? And what is the difference between "multithreaded ingestion" and "batch ingestion"?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

I believe "multi-threaded ingestion" refers to "Index Parallelization" in which indexer manages multiple piplelines to allow parallel event parsing. See these for more details on the same

https://conf.splunk.com/files/2016/slides/harnessing-performance-and-scalability-with-parallelizatio... (Slide 11 onwards)

http://docs.splunk.com/Documentation/Forwarder/7.1.1/Forwarder/Configureaforwardertohandlemultiplepi...
https://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Pipelinesets

View solution in original post

Moderator
Moderator

Hi @castillorm

My name is Anam Siddique and I am the Community Content Specialist for Splunk Answers. Please accept the appropriate answer that worked for you so other members of the community can benefit from it. If none of the answers have worked for you so far please post further comments so someone can help you.

Thanks

0 Karma

Explorer

This may also provide some clarification on multithread ingest/parallel indexes: https://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Pipelinesets

0 Karma

Contributor

Esteemed Legend

Some paths through the input queue are single-threaded. For example, if you are forwarding compressed files you should be aware that the AEQ (AKA AQ, Archive Queue) handler is single-threaded and becomes a HUGE bottleneck with even small numbers of *.zip files. I once had ~30 forwarder instances installed on a single UF just to handle *.zip files coming in.

0 Karma

SplunkTrust
SplunkTrust

I believe "multi-threaded ingestion" refers to "Index Parallelization" in which indexer manages multiple piplelines to allow parallel event parsing. See these for more details on the same

https://conf.splunk.com/files/2016/slides/harnessing-performance-and-scalability-with-parallelizatio... (Slide 11 onwards)

http://docs.splunk.com/Documentation/Forwarder/7.1.1/Forwarder/Configureaforwardertohandlemultiplepi...
https://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Pipelinesets

View solution in original post

Champion

Yes, Its index parellelization.

————————————
If this helps, give a like below.
0 Karma