https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409672#M72636 <P>Clearly I did not dig around enough. Thank you so much for your help! </P> Thu, 16 Aug 2018 17:13:37 GMT nealw 2018-08-16T17:13:37Z https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409670#M72634 <P>Hello, </P> <P>I googled around for similar questions but could not find anything, so I'm sorry if this question has already been asked before. If i want to index large amounts of data using multiple forwarders, is there some way where i can configure the various forwarders to act in a distributed fashion? I know of managing pipelines for index parallelization (<A href="https://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Pipelinesets">https://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Pipelinesets</A>), but that still does not quite solve the issue. </P> <P>What do people in general do to solve such a problem? Thank you! </P> Thu, 16 Aug 2018 00:55:34 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409670#M72634 nealw 2018-08-16T00:55:34Z https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409671#M72635 <P>hello there,<BR /> very detailed answer by @mmodestino_splunk here:<BR /> <A href="https://answers.splunk.com/answers/521464/how-to-run-multiple-universal-forwarders-on-a-sing.html">https://answers.splunk.com/answers/521464/how-to-run-multiple-universal-forwarders-on-a-sing.html</A></P> <P>hope it helps</P> Thu, 16 Aug 2018 03:25:21 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409671#M72635 adonio 2018-08-16T03:25:21Z https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409672#M72636 <P>Clearly I did not dig around enough. Thank you so much for your help! </P> Thu, 16 Aug 2018 17:13:37 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409672#M72636 nealw 2018-08-16T17:13:37Z https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409673#M72637 <P>Right @adonio, so we can install multiple forwarders on the same physical machine which is nice. The major question was - </P> <P>-- is there some way where i can configure the various forwarders to act in a distributed fashion?</P> <P>How do we do that - <STRONG>distributed fashion</STRONG>?</P> Fri, 17 Aug 2018 00:49:10 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409673#M72637 ddrillic 2018-08-17T00:49:10Z https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409674#M72638 <P>@ddrillic, indeed, it was late in the night when answered the question. <BR /> can you elaborate on the question? what does "distributed fashion" means?</P> Fri, 17 Aug 2018 03:06:02 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409674#M72638 adonio 2018-08-17T03:06:02Z https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409675#M72639 <P>I think, or at least this was what I was trying to get at - more like, in parallel. There are some ways of installing multiple universal forwarders and forwarding data to an intermediate forwarder and filtering out duplicate data, or just assigning the universal forwarders different directories to forward data from. But can we deploy universal forwarders in such a way that they coordinate and work in parallel? More like how multiple pipelines work, for index parallelization. <BR /> Thanks!</P> Sat, 18 Aug 2018 00:02:15 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409675#M72639 nealw 2018-08-18T00:02:15Z https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409676#M72640 <P>i hope i understand where you are getting at, <BR /> the forwarders are set to do what they being told to do. assuming you have 2 forwarders on a single machine, and these forwarders are monitoring 2 unique directories, forwarder A monitors directory A and forwarder B, monitors directory B. let say, now there is double of the amount of local data being written to directory A and no more data being written to directory B, I am not aware of (and i am 99% sure there isnt) a way forwarder B will "help" forwarder A and share the monitoring load of directory A<BR /> @nealw, hope this answers your question</P> Sat, 18 Aug 2018 12:24:46 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409676#M72640 adonio 2018-08-18T12:24:46Z https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409677#M72641 <P>Ah, i see. Well, that does answer what I was trying to get at. Thanks a lot! </P> Mon, 20 Aug 2018 18:01:10 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-index-data-with-multiple-forwarders-on-the-same-host/m-p/409677#M72641 nealw 2018-08-20T18:01:10Z