https://community.splunk.com/t5/Getting-Data-In/How-to-prevent-large-lookups-from-being-replicated-to-Yarn/m-p/235255#M45813 <P>We have a user who has created a large csv lookup file (600 Mb). It seems that this file is being replicated to every Yarn server to it's /tmp directory (/tmp/splunk/nodename/splunk/var/run/searchpeers) apparently as every search is executed. There are multiple copies of this bundle. This is filling the /tmp directory and causing a major problem, it also slows every search as this file has to be copied before the search begins to execute. </P> <P>We have specified the following in the hunk servers distsearch.conf to no effect: <BR /> [replicationBlacklist] <BR /> Everything = Servers.csv </P> <P>How do we block this file being copied with every search? <BR /> Can the file be moved into an HDFS directory instead? <BR /> Can it be cached so that it doesn't need to be replicated with every search? </P> Mon, 09 May 2016 17:07:27 GMT tsunamii 2016-05-09T17:07:27Z https://community.splunk.com/t5/Getting-Data-In/How-to-prevent-large-lookups-from-being-replicated-to-Yarn/m-p/235255#M45813 <P>We have a user who has created a large csv lookup file (600 Mb). It seems that this file is being replicated to every Yarn server to it's /tmp directory (/tmp/splunk/nodename/splunk/var/run/searchpeers) apparently as every search is executed. There are multiple copies of this bundle. This is filling the /tmp directory and causing a major problem, it also slows every search as this file has to be copied before the search begins to execute. </P> <P>We have specified the following in the hunk servers distsearch.conf to no effect: <BR /> [replicationBlacklist] <BR /> Everything = Servers.csv </P> <P>How do we block this file being copied with every search? <BR /> Can the file be moved into an HDFS directory instead? <BR /> Can it be cached so that it doesn't need to be replicated with every search? </P> Mon, 09 May 2016 17:07:27 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-prevent-large-lookups-from-being-replicated-to-Yarn/m-p/235255#M45813 tsunamii 2016-05-09T17:07:27Z https://community.splunk.com/t5/Getting-Data-In/How-to-prevent-large-lookups-from-being-replicated-to-Yarn/m-p/235256#M45814 <P>To make sure the Table is not being copied every time:<BR /> vix.splunk.setup.onsearch = 0 (default is 1)<BR /> ** However, that means nothing will be copied to the data nodes. So you may want to turn it off only after the first run.</P> <P>To make sure you have lots of copies of the table so that bundle replication happens fast:<BR /> vix.splunk.setup.bundle.replication = 20 (default 3)</P> Mon, 09 May 2016 18:24:30 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-prevent-large-lookups-from-being-replicated-to-Yarn/m-p/235256#M45814 rdagan_splunk 2016-05-09T18:24:30Z https://community.splunk.com/t5/Getting-Data-In/How-to-prevent-large-lookups-from-being-replicated-to-Yarn/m-p/235257#M45815 <P>Assuming that this is being used as a <CODE>lookup</CODE> file, you can specify that the lookup happens only on the Search Head by adding the <CODE>local=t</CODE> parameter as in <CODE>... | lookup local=t myLookup ...</CODE>. This will prevent it from being included in the bundle (replication). The downside is that if the output fields of the lookup are used to qualify the search at any point, you will lose the benefits of having this work being map-reduced and happening on the Indexers; instead it will all happen on the Search Head.</P> Tue, 10 May 2016 04:24:18 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-prevent-large-lookups-from-being-replicated-to-Yarn/m-p/235257#M45815 woodcock 2016-05-10T04:24:18Z