topic Re: What is multithreaded ingestion? in Getting Data In

What is multithreaded ingestion?

castillorm — Mon, 23 Jul 2018 19:23:47 GMT

Does anyone know what is meant by the term "multithreaded ingestion"? And what is the difference between "multithreaded ingestion" and "batch ingestion"?

Re: What is multithreaded ingestion?

somesoni2 — Mon, 23 Jul 2018 20:29:09 GMT

I believe "multi-threaded ingestion" refers to "Index Parallelization" in which indexer manages multiple piplelines to allow parallel event parsing. See these for more details on the same

https://conf.splunk.com/files/2016/slides/harnessing-performance-and-scalability-with-parallelization.pdf (Slide 11 onwards)

http://docs.splunk.com/Documentation/Forwarder/7.1.1/Forwarder/Configureaforwardertohandlemultiplepipelinesets
https://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Pipelinesets

Re: What is multithreaded ingestion?

thambisetty — Fri, 27 Jul 2018 17:49:36 GMT

Yes, Its index parellelization.

Re: What is multithreaded ingestion?

woodcock — Fri, 03 Aug 2018 15:15:11 GMT

Some paths through the input queue are single-threaded. For example, if you are forwarding compressed files you should be aware that the AEQ (AKA AQ, Archive Queue) handler is single-threaded and becomes a HUGE bottleneck with even small numbers of *.zip files. I once had ~30 forwarder instances installed on a single UF just to handle *.zip files coming in.

Re: What is multithreaded ingestion?

felipesewaybric — Fri, 03 Aug 2018 15:30:24 GMT

check this answer:
https://answers.splunk.com/answers/12027/single-multi-threading-cpu.html

Re: What is multithreaded ingestion?

kariras06 — Fri, 03 Aug 2018 20:02:47 GMT

This may also provide some clarification on multithread ingest/parallel indexes: https://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Pipelinesets

Re: What is multithreaded ingestion?

Anam — Thu, 09 Aug 2018 18:30:56 GMT

Hi @castillorm

My name is Anam Siddique and I am the Community Content Specialist for Splunk Answers. Please accept the appropriate answer that worked for you so other members of the community can benefit from it. If none of the answers have worked for you so far please post further comments so someone can help you.

Thanks