Splunk Memory error with YARN

[aaron_harris]

When running a search in splunk such as 'index=syslog date_hour=12' we get the below error to do with memory configurations, however amendments to the Virtual Index provider's memory settings don't seem to have any affect.

com.splunk.mr.JobStartException: Failed to start MapReduce job. Please consult search.log for more information. Message: [ Failed to start MapReduce job,  name=SPLK_splunk_1464947571.2_0 ] and [ org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < ​0, or requested memory > max configured, requestedMemory=1536, maxMemory=512

Indexes.conf configuration file contains below

[provider:test]
vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-hy2.jar
vix.env.HADOOP_HOME = /usr/hdp/current/hadoop-client
vix.env.JAVA_HOME = /usr/lib/jvm/jre-1.8.0
vix.family = hadoop
vix.fs.default.name = hdfs://{{hdfs_namenode_fqdn}}:8020
vix.hadoop.security.authorization = 0
vix.mapreduce.framework.name = yarn
vix.yarn.resourcemanager.address={{hdfs_resourcemanager_fqdn}}:8050
vix.yarn.resourcemanager.scheduler.address={{hdfs_resourcemanager​_fqdn}}:8030
vix.mapred.job.map.memory.mb = 256
vix.mapred.job.reduce.memory.mb = 128​
vix.mapreduce.map.memory.mb=256
vix.mapreduce.reduce.memory.mb=128
vix.output.buckets.max.network.bandwidth = 0
vix.splunk.home.hdfs = /hunk/virt_indexes

[syslog]
vix.input.1.path = /syslog/...
vix.provider = test    

[rdagan_splunk]

Since you are using Yarn, try and change the value of the flag vix.mapreduce.map.memory.mb instead of vix.mapred.job.map.memory.mb
Hadoop version 1 uses mapred and Hadoop version 2 (yarn) uses mapreduce

[aaron_harris]

Thanks for the response, have tried adding the following lines to the indexes.conf file but still get the same error when searching in Splunk.

vix.mapreduce.map.memory.mb=256
vix.mapreduce.reduce.memory.mb=128