topic Batching gzipped files residing in 4 directories into Splunk, is there a way to run parallel batches on a Splunk 6.2.6 Linux universal forwarder? in Getting Data In

Batching gzipped files residing in 4 directories into Splunk, is there a way to run parallel batches on a Splunk 6.2.6 Linux universal forwarder?

lisaac — Wed, 28 Oct 2015 16:39:42 GMT

I am batching gzipped files into Splunk. The files reside in 4 directories. Splunk, per splunkd.log, appears to be reading only the files in the first batch statement. Is there a way to run parallel batches?

I have a Linux 64 bit Universal Forwarder with Splunk 6.2.6. I have set maxKBPS to 0 in limits.conf, and I have reniced the Splunk UF prdata2sses to a priority of -20 on the Linux VM.

I have batch statements listed as follows:

[batch:///leroylogs2/multicast/archive/data11/*PROD*.gz]
[batch:///leroylogs2/multicast/archive/data12/*PROD*.gz]
[batch:///leroylogs2/multicast/archive/data21/*PROD*.gz]
[batch:///leroylogs2/multicast/archive/data22/*PROD*.gz]
[batch:///leroylogs2/multicast/archive/data11/*CAS*.gz]
[batch:///leroylogs2/multicast/archive/data12/*CAS*.gz]
[batch:///leroylogs2/multicast/archive/data21/*CAS*.gz]
[batch:///leroylogs2/multicast/archive/data22/*CAS*.gz]

10-28-2015 09:37:37.372 -0700 INFO  ArchiveProcessor - Finished processing file '/leroylogs2/multicast/archive/data11/2015-08-29-08_30-PRODtrans_svs.log.gz', removing from stats
10-28-2015 09:37:37.433 -0700 INFO  ArchiveProcessor - handling file=/leroylogs2/multicast/archive/data11/2015-08-29-07_10-CAStrans_svs.log.gz
10-28-2015 09:37:37.434 -0700 INFO  ArchiveProcessor - reading path=/leroylogs2/multicast/archive/data11/2015-08-29-07_10-CAStrans_svs.log.gz (seek=0 len=32496625)
10-28-2015 09:37:37.551 -0700 WARN  TcpOutputProc - The event is missing source information. Event :
10-28-2015 09:37:38.655 -0700 ERROR ArchiveContext - From archive='/leroylogs2/multicast/archive/data11/2015-08-29-07_10-CAStrans_svs.log.gz':  gzip: stdout: Broken pipe

Re: Batching gzipped files residing in 4 directories into Splunk, is there a way to run parallel batches on a Splunk 6.2.6 Linux universal forwarder?

lisaac — Wed, 28 Oct 2015 17:30:49 GMT

I suppose that I could run 2 UFs on the same host, but I would prefer to skip this approach.

Re: Batching gzipped files residing in 4 directories into Splunk, is there a way to run parallel batches on a Splunk 6.2.6 Linux universal forwarder?

anekkanti_splun — Fri, 30 Oct 2015 18:47:43 GMT

In Pre 6.3: The only way to read archives/files in parallel is by spawning multiple instances of splunk on the forwarder.
Splunk 6.3 release has a new feature where you can spawn multiple ingestion pipelines, where each pipeline can read one archive/file independently. So essentially with multiple ingestion pipelines, splunk will read multiple archives/files in parallel.
Documentation: http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/Pipelinesets