How to resolve authentication issue with virtual provider/index?

[mdsnmss]

I'm working on making a connection to a virtual provider with a virtual index I have confirmed with some test data. I've enabled debug for the provider and executed some searches. The search just executes and says "No results found" but looking at the search.log shows some more in depth analysis of what is occurring. These events are below. One to highlight is

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled. Available:[TOKEN].

I've confirmed these settings are on the Hadoop provider in core-site.xml:

<configuration>
<property>
      <name>hadoop.security.authentication</name>
      <value>simple</value>
    </property>
    <property>
      <name>hadoop.security.authorization</name>
      <value>false</value>
    </property>
</configuration>

I've added it to the core-site.xml on my search head as well but am not sure if/how I need to reload it. Anyone have any ideas? I just have a test directory for my virtual index to hit I have confirmed with some simple linux_secure events.

04-18-2019 14:45:04.268 INFO  ERP.virtual_provider -  SplunkMR - starting, version=6.2 ...
04-18-2019 14:45:05.038 INFO  ERP.virtual_provider -  Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
04-18-2019 14:45:05.048 INFO  ERP.virtual_provider -  SplunkMR - Setting custom jars=file:/opt/splunk/bin/jars/thirdparty/common/avro-1.7.7.jar,file:/opt/splunk/bin/jars/thirdparty/common/avro-mapred-1.7.7.jar,file:/opt/splunk/bin/jars/thirdparty/common/commons-compress-1.10.jar,file:/opt/splunk/bin/jars/thirdparty/common/commons-io-2.4.jar,file:/opt/splunk/bin/jars/thirdparty/common/libfb303-0.9.2.jar,file:/opt/splunk/bin/jars/thirdparty/common/parquet-hive-bundle-1.6.0.jar,file:/opt/splunk/bin/jars/thirdparty/common/snappy-java-1.1.1.7.jar,file:/opt/splunk/bin/jars/thirdparty/hive_1_2/hive-exec-1.2.1.jar,file:/opt/splunk/bin/jars/thirdparty/hive_1_2/hive-metastore-1.2.1.jar,file:/opt/splunk/bin/jars/thirdparty/hive_1_2/hive-serde-1.2.1.jar
04-18-2019 14:45:05.093 INFO  ERP.virtual_provider -  SplunkMR - Configured to use installed splunk package. Ensuring package exists
04-18-2019 14:45:05.099 INFO  ERP.virtual_provider -  SplunkMR - Setting configuration to use local Splunk package: /opt/splunk/splunk-7.0.2-03bbabbd5c0f-linux-2.6-x86_64.tgz
04-18-2019 14:45:05.158 DEBUG ERP.virtual_provider -  VirtualIndex$VIXPathSpecifier - VIXPathSpecifier globpath=/tmp/splunk/secure/*, accept=null, ignore=null
04-18-2019 14:45:05.165 DEBUG ERP.virtual_provider -  SplunkBaseMapper - RecordReader list, index=null, inputId=null, list=SplunkJournalRecordReader,ValueAvroRecordReader,SimpleCSVRecordReader,SequenceFileRecordReader
04-18-2019 14:45:05.183 INFO  ERP.virtual_provider -  SplunkMR$SearchHandler - Reduce search: null
04-18-2019 14:45:05.183 INFO  ERP.virtual_provider -  SplunkMR$SearchHandler - Search mode: stream
04-18-2019 14:45:05.183 DEBUG ERP.virtual_provider -  SplunkMR$SearchHandler - parsed _keySet=[index::virtual_index, source::/tmp/splunk/secure]
04-18-2019 14:45:05.183 INFO  ERP.virtual_provider -  SplunkMR$SearchHandler - setting requiredFields=*
04-18-2019 14:45:05.231 DEBUG ERP.virtual_provider -  MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
04-18-2019 14:45:05.237 DEBUG ERP.virtual_provider -  MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
04-18-2019 14:45:05.237 DEBUG ERP.virtual_provider -  MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
04-18-2019 14:45:05.238 DEBUG ERP.virtual_provider -  MetricsSystemImpl - UgiMetrics, User and group related metrics
04-18-2019 14:45:05.323 DEBUG ERP.virtual_provider -  KerberosName - Kerberos krb5 configuration not found, setting default realm to empty
04-18-2019 14:45:05.323 DEBUG ERP.virtual_provider -  Groups -  Creating new Groups object
04-18-2019 14:45:05.324 DEBUG ERP.virtual_provider -  NativeCodeLoader - Trying to load the custom-built native-hadoop library...
04-18-2019 14:45:05.324 DEBUG ERP.virtual_provider -  NativeCodeLoader - Loaded the native-hadoop library
04-18-2019 14:45:05.325 DEBUG ERP.virtual_provider -  JniBasedUnixGroupsMapping - Using JniBasedUnixGroupsMapping for Group resolution
04-18-2019 14:45:05.325 DEBUG ERP.virtual_provider -  JniBasedUnixGroupsMappingWithFallback - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
04-18-2019 14:45:05.451 DEBUG ERP.virtual_provider -  Groups - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
04-18-2019 14:45:05.455 DEBUG ERP.virtual_provider -  UserGroupInformation$HadoopLoginModule - hadoop login
04-18-2019 14:45:05.455 DEBUG ERP.virtual_provider -  UserGroupInformation$HadoopLoginModule - hadoop login commit
04-18-2019 14:45:05.462 DEBUG ERP.virtual_provider -  UserGroupInformation$HadoopLoginModule - using local user:UnixPrincipal: splunk
04-18-2019 14:45:05.462 DEBUG ERP.virtual_provider -  UserGroupInformation$HadoopLoginModule - Using user: "UnixPrincipal: splunk" with name splunk
04-18-2019 14:45:05.462 DEBUG ERP.virtual_provider -  UserGroupInformation$HadoopLoginModule - User entry: "splunk"
04-18-2019 14:45:05.463 DEBUG ERP.virtual_provider -  UserGroupInformation - UGI loginUser:splunk (auth:SIMPLE)
04-18-2019 14:45:05.649 DEBUG ERP.virtual_provider -  DFSClient$Conf - dfs.client.use.legacy.blockreader.local = false
04-18-2019 14:45:05.649 DEBUG ERP.virtual_provider -  DFSClient$Conf - dfs.client.read.shortcircuit = false
04-18-2019 14:45:05.652 DEBUG ERP.virtual_provider -  DFSClient$Conf - dfs.client.domain.socket.data.traffic = false
04-18-2019 14:45:05.652 DEBUG ERP.virtual_provider -  DFSClient$Conf - dfs.domain.socket.path = 
04-18-2019 14:45:05.688 DEBUG ERP.virtual_provider -  RetryUtils - multipleLinearRandomRetry = null
04-18-2019 14:45:05.707 DEBUG ERP.virtual_provider -  Server - rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@dc9876b
04-18-2019 14:45:06.068 DEBUG ERP.virtual_provider -  DomainSocketWatcher$2 - org.apache.hadoop.net.unix.DomainSocketWatcher$2@3ffc7c56: starting with interruptCheckPeriodMs = 60000
04-18-2019 14:45:06.081 DEBUG ERP.virtual_provider -  DomainSocketFactory - Both short-circuit local reads and UNIX domain socket are disabled.
04-18-2019 14:45:06.090 DEBUG ERP.virtual_provider -  DataTransferSaslUtil - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
04-18-2019 14:45:06.090 INFO  ERP.virtual_provider -  SplunkMR$SearchHandler - Created filesystem object, elapsed_ms=910
04-18-2019 14:45:06.198 DEBUG ERP.virtual_provider -  NativeAzureFileSystem - finalize() called.
04-18-2019 14:45:06.199 DEBUG ERP.virtual_provider -  NativeAzureFileSystem - finalize() called.
04-18-2019 14:45:06.352 DEBUG ERP.virtual_provider -  SearchController - stopOnMaxHadoopNodesExceedingLicense ...
04-18-2019 14:45:06.374 DEBUG ERP.virtual_provider -  SSLUtil$SSLv3SocketFactory - Keeping without change enabled SSL protocols: ['TLSv1', 'TLSv1.1', 'TLSv1.2']
04-18-2019 14:45:06.507 INFO  ERP.virtual_provider -  Configuration - mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
04-18-2019 14:45:06.519 DEBUG ERP.virtual_provider -  VixSplitGenerator - Generating splits for index=virtual_index
04-18-2019 14:45:06.527 INFO  ERP.virtual_provider -  VirtualIndex$Splitter - generateSplits started, vix.name=virtual_index ... 
04-18-2019 14:45:06.527 DEBUG ERP.virtual_provider -  VirtualIndex$Splitter - using SplitGenerator class=com.splunk.mr.input.FileSplitGenerator, to split files in index=virtual_index inputId=1
04-18-2019 14:45:06.527 DEBUG ERP.virtual_provider -  Cluster - Trying ClientProtocolProvider : org.apache.hadoop.mapred.LocalClientProtocolProvider
04-18-2019 14:45:06.527 DEBUG ERP.virtual_provider -  Cluster - Cannot pick org.apache.hadoop.mapred.LocalClientProtocolProvider as the ClientProtocolProvider - returned null protocol
04-18-2019 14:45:06.527 DEBUG ERP.virtual_provider -  Cluster - Trying ClientProtocolProvider : org.apache.hadoop.mapred.YarnClientProtocolProvider
04-18-2019 14:45:06.545 DEBUG ERP.virtual_provider -  AbstractService - Service: org.apache.hadoop.mapred.ResourceMgrDelegate entered state INITED
04-18-2019 14:45:06.548 DEBUG ERP.virtual_provider -  AbstractService - Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state INITED
04-18-2019 14:45:06.594 INFO  ERP.virtual_provider -  RMProxy - Connecting to ResourceManager at /<ip>:<port>
04-18-2019 14:45:06.594 DEBUG ERP.virtual_provider -  UserGroupInformation - PrivilegedAction as:splunk (auth:SIMPLE) from:org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:136)
04-18-2019 14:45:06.598 DEBUG ERP.virtual_provider -  YarnRPC - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
04-18-2019 14:45:06.599 DEBUG ERP.virtual_provider -  HadoopYarnProtoRPC - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ApplicationClientProtocol
04-18-2019 14:45:06.732 DEBUG ERP.virtual_provider -  AbstractService - Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl is started
04-18-2019 14:45:06.732 DEBUG ERP.virtual_provider -  AbstractService - Service org.apache.hadoop.mapred.ResourceMgrDelegate is started
04-18-2019 14:45:06.753 DEBUG ERP.virtual_provider -  ProtobufRpcEngine$Invoker - Call: getFileInfo took 208ms
04-18-2019 14:45:06.765 DEBUG ERP.virtual_provider -  UserGroupInformation - PrivilegedAction as:splunk (auth:SIMPLE) from:org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:331)
04-18-2019 14:45:06.770 DEBUG ERP.virtual_provider -  DFSClient$Conf - dfs.client.use.legacy.blockreader.local = false
04-18-2019 14:45:06.770 DEBUG ERP.virtual_provider -  DFSClient$Conf - dfs.client.read.shortcircuit = false
04-18-2019 14:45:06.770 DEBUG ERP.virtual_provider -  DFSClient$Conf - dfs.client.domain.socket.data.traffic = false
04-18-2019 14:45:06.770 DEBUG ERP.virtual_provider -  DFSClient$Conf - dfs.domain.socket.path = 
04-18-2019 14:45:06.771 DEBUG ERP.virtual_provider -  RetryUtils - multipleLinearRandomRetry = null
04-18-2019 14:45:06.771 DEBUG ERP.virtual_provider -  DataTransferSaslUtil - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
04-18-2019 14:45:06.772 DEBUG ERP.virtual_provider -  Cluster - Picked org.apache.hadoop.mapred.YarnClientProtocolProvider as the ClientProtocolProvider
04-18-2019 14:45:06.773 DEBUG ERP.virtual_provider -  UserGroupInformation - PrivilegedAction as:splunk (auth:SIMPLE) from:org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:802)
04-18-2019 14:45:06.804 DEBUG ERP.virtual_provider -  VirtualIndex$VIXPathSpecifier - readFromDM=false, writeToDM=false
04-18-2019 14:45:06.892 DEBUG ERP.virtual_provider -  ProtobufRpcEngine$Invoker - Call: getListing took 74ms
04-18-2019 14:45:06.931 DEBUG ERP.virtual_provider -  UserGroupInformation - PrivilegedActionException as:splunk (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN]
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -  ClusterInfoLogger - Exception thrown while logging cluster info
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -  org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN]
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:209)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at java.lang.reflect.Method.invoke(Method.java:498)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at com.sun.proxy.$Proxy17.getClusterMetrics(Unknown Source)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:487)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.mapred.ResourceMgrDelegate.getClusterMetrics(ResourceMgrDelegate.java:151)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.mapred.YARNRunner.getClusterMetrics(YARNRunner.java:179)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.mapreduce.Cluster.getClusterStatus(Cluster.java:247)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.mapred.JobClient$4.run(JobClient.java:804)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.mapred.JobClient$4.run(JobClient.java:802)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at java.security.AccessController.doPrivileged(Native Method)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at javax.security.auth.Subject.doAs(Subject.java:422)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:802)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at com.splunk.mr.ClusterInfo.<init>(ClusterInfo.java:23)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at com.splunk.mr.ClusterInfo.getInstance(ClusterInfo.java:32)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at com.splunk.mr.ClusterInfoLogger.run(ClusterInfoLogger.java:69)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -  Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled.  Available:[TOKEN]
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.ipc.Client.call(Client.java:1475)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.ipc.Client.call(Client.java:1412)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at com.sun.proxy.$Proxy16.getClusterMetrics(Unknown Source)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:206)
04-18-2019 14:45:06.934 WARN  ERP.virtual_provider -    ... 20 more
04-18-2019 14:45:06.968 DEBUG ERP.virtual_provider -  ProtobufRpcEngine$Invoker - Call: getFileInfo took 72ms
04-18-2019 14:45:06.971 INFO  ERP.virtual_provider -  VirtualIndex - generateSplits done, vix.name=virtual_index, files.total=0, files.time.filtered=0, files.search.filtered=0, files.mr=0, elapsed=453ms
04-18-2019 14:45:06.971 INFO  ERP.virtual_provider -  SplunkMR$SearchHandler - The search couldn't find any matching data
04-18-2019 14:45:06.973 INFO  ERP.virtual_provider -  SplunkMR - finishing, version=6.2 ...
04-18-2019 14:45:07.339 INFO  ERPSearchResultCollector - ERP peer=virtual_provider is done reading search results.

[mdsnmss · Accepted solution]

It was actually a combination of issues. Ports on the Resource Manager and Resource Scheduler were reversed for our provider. For the virtual index we all had it set to recurse on a file when was pointing directly at a file so it needed to be unchecked for that component.

[mdsnmss]

Issues appear to begin at line 70.