What is multithreaded ingestion?

[castillorm]

Does anyone know what is meant by the term "multithreaded ingestion"? And what is the difference between "multithreaded ingestion" and "batch ingestion"?

[Anam]

Hi @castillorm

My name is Anam Siddique and I am the Community Content Specialist for Splunk Answers. Please accept the appropriate answer that worked for you so other members of the community can benefit from it. If none of the answers have worked for you so far please post further comments so someone can help you.

Thanks

[kariras06]

This may also provide some clarification on multithread ingest/parallel indexes: https://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Pipelinesets

[felipesewaybric]

check this answer:
https://answers.splunk.com/answers/12027/single-multi-threading-cpu.html

[woodcock]

Some paths through the input queue are single-threaded. For example, if you are forwarding compressed files you should be aware that the AEQ (AKA AQ, Archive Queue) handler is single-threaded and becomes a HUGE bottleneck with even small numbers of *.zip files. I once had ~30 forwarder instances installed on a single UF just to handle *.zip files coming in.

[somesoni2 · Accepted solution]

I believe "multi-threaded ingestion" refers to "Index Parallelization" in which indexer manages multiple piplelines to allow parallel event parsing. See these for more details on the same

https://conf.splunk.com/files/2016/slides/harnessing-performance-and-scalability-with-parallelizatio... (Slide 11 onwards)

http://docs.splunk.com/Documentation/Forwarder/7.1.1/Forwarder/Configureaforwardertohandlemultiplepi...
https://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Pipelinesets

[thambisetty]

Yes, Its index parellelization.