topic Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255 in Splunk Dev

Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Tue, 29 Sep 2020 19:37:11 GMT

We have the MapR filesystem and Hunk on the same node, have set up a Provider and Virtual and Im getting an error when trying to run a query other than a basic index search.

"index=test" produces correct results
"index=test keyword" produces the error below

Also see screenshot attachments of Provider and VI config.

Error from search.log:

05-23-2018 20:49:13.581 ERROR ERP.maproly - Caused by: java.lang.RuntimeException: summary_id did not exist in search info: {_tz=### SERIALIZED TIMEZONE FORMAT 1.0;C0;Y0 NW 55 54 43;$, now=1527108550.000000000, _sid=1527108550.15, site=default, _api_et=1527019200.000000000, _api_lt=1527108550.000000000, _dsi_id=0, _keySet=dghsd index::test1, _ppc.bs=$SPLUNK_ETC, _search=search index=test1 dghsd, _shp_id=C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6, _endTime=1527108550.000000000, _ppc.app=search, read_raw=1, realtime=0, _countMap=duration.command.search.expand_search;39;duration.command.search.parse_directives;0;duration.dispatch.evaluate.search;54;duration.startup.configuration;11;duration.startup.handoff;3;invocations.command.search.expand_search;1;invocations.command.search.parse_directives;1;invocations.dispatch.evaluate.search;1;invocations.startup.configuration;1;invocations.startup.handoff;1;, _ppc.user=admin, check_dangerous_command=0, _default_group=, generation_id=0, _bundle_version=0, indexed_realtime=0, search_can_be_event_type=1, indexed_realtime_offset=0, kv_store_settings=hosts;127.0.0.1:8191\;;local;127.0.0.1:8191;read_preference;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6;replica_set_name;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6;status;ready;, _timeline_events_preview=0, is_cluster_slave=0, internal_only=0, is_batch_mode=0, _remote_search=search (index=test1 dghsd) | fields keepcolorder=t "" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server", summary_stopped=0, _search_metrics={"ConsideredBuckets":0,"EliminatedBuckets":0,"ConsideredEvents":0,"TotalSlicesInBuckets":0,"DecompressedSlices":0,"FieldMetadata_Events":"","Partition":{}}, _is_summary_index=0, _search_StartUp_Spent=0, _is_keepalive=0, _is_scheduled=0, _splunkd_port=8089, _is_export=0, _is_remote=0, _maxevents=0, _search_et=1527019200.000000000, _search_lt=1527108550.000000000, _startTime=1527019200.000000000, _timestamp=1527108550.251636000, is_saved_search=0, is_remote_sorted=0, _search_StartTime=1527108550.250084000, remote_log_download_mode=disabledSavedSearches, kv_store_additional_settings=hosts_guids;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6\;;, _rt_batch_retry=0, _auth_token=8cntqHuq0Rb0Lz3T^YcThKI7mBeHBy4ki7SPCQHDHCuMQq1haa4BENOHDqd43diGvYDkRlyNuR6xs1eUwYfPE4PBO1IeTwbkxIAG2JxOpUIpE^IOBBwklXUWaqa, _drop_count=0, _provenance=UI:Search, _scan_count=0, is_shc_mode=0, rt_backfill=0, sample_seed=0, _bs_thread_count=1, _retry_count=0, _splunkd_uri=https://127.0.0.1:8089, replay_speed=0, _exported_results=0, sample_ratio=1, summary_mode=none, _query_finished=1, _optional_fields_json={}, enable_event_stream=1, _splunkd_protocol=https, _read_buckets_since_startup=0, _bs_pipeline_identifier=0, _request_finalization=0}
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.getSummaryId(SplunkMR.java:507)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.executeMapReduce(SplunkMR.java:1359)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1067)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.run(SplunkMR.java:1551)
05-23-2018 20:49:13.581 ERROR ERP.maproly - ... 3 more
05-23-2018 20:49:13.597 INFO ERP.maproly - SplunkMR - finishing, version=6.2 ...
05-23-2018 20:49:13.597 INFO ERP.maproly - DispatchReaper - Skip dispatch reaping, top level HDFS dispatch dir=/user/root/splunk/splunkmr/dispatch does not exist.
05-23-2018 20:49:13.621 ERROR ERP.maproly - Error while invoking command: /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

rdagan_splunk — Thu, 24 May 2018 14:04:13 GMT

It looks as if you are using TaskTracker and not Yarn.
Change your settings to Yarn and point to Yarn resource Manager instead of Task Tracker.
You are using Hadoop 2.7, which default to Yarn.
In addition, your path to data in HDFS looks wrong. Normally all you need is /user/username

To test if you have the right location of the file in HDFS, I will recommend for you to try this command from CLI:
hadoop fs -ls maprfs:///user/mapr

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Thu, 24 May 2018 14:46:45 GMT

Thank you for your reply. Do you know how I can discover the path to my Yarn Resource Manager?

Its interesting you say that the location of the file in HDFS is wrong because that is definitely where it is. I am also able to bring up that data via that directory with a simple query of "index=test" so it is finding it.
When I run your command I get:

[root@hadoop-s1 elloyd]# hadoop fs -ls maprfs:///user/mapr
Found 22 items
drwxr-xr-x - root root 1 2018-05-23 21:57 maprfs:///user/mapr/2018
drwxr-xr-x - mapr mapr 1 2018-05-10 00:56 maprfs:///user/mapr/drill
-rw-r--r-- 3 root root 0 2018-05-24 14:00 maprfs:///user/mapr/dzl.log
-rw-r--r-- 3 root root 0 2018-05-23 22:00 maprfs:///user/mapr/dzl.log-20180523220001
-rw-r--r-- 3 root root 0 2018-05-23 23:00 maprfs:///user/mapr/dzl.log-20180523230001
-rw-r--r-- 3 root root 0 2018-05-24 00:00 maprfs:///user/mapr/dzl.log-20180524000001
-rw-r--r-- 3 root root 0 2018-05-24 01:00 maprfs:///user/mapr/dzl.log-20180524010002
-rw-r--r-- 3 root root 0 2018-05-24 02:00 maprfs:///user/mapr/dzl.log-20180524020001
-rw-r--r-- 3 root root 0 2018-05-24 03:00 maprfs:///user/mapr/dzl.log-20180524030001
-rw-r--r-- 3 root root 0 2018-05-24 04:00 maprfs:///user/mapr/dzl.log-20180524040001
-rw-r--r-- 3 root root 0 2018-05-24 05:00 maprfs:///user/mapr/dzl.log-20180524050002
-rw-r--r-- 3 root root 0 2018-05-24 06:00 maprfs:///user/mapr/dzl.log-20180524060002
-rw-r--r-- 3 root root 0 2018-05-24 07:00 maprfs:///user/mapr/dzl.log-20180524070001
-rw-r--r-- 3 root root 0 2018-05-24 08:00 maprfs:///user/mapr/dzl.log-20180524080001
-rw-r--r-- 3 root root 0 2018-05-24 09:00 maprfs:///user/mapr/dzl.log-20180524090001
-rw-r--r-- 3 root root 0 2018-05-24 10:00 maprfs:///user/mapr/dzl.log-20180524100001
-rw-r--r-- 3 root root 0 2018-05-24 11:00 maprfs:///user/mapr/dzl.log-20180524110002
-rw-r--r-- 3 root root 0 2018-05-24 12:00 maprfs:///user/mapr/dzl.log-20180524120001
-rw-r--r-- 3 root root 0 2018-05-24 13:00 maprfs:///user/mapr/dzl.log-20180524130001
-rw-r--r-- 3 root root 0 2018-05-24 14:00 maprfs:///user/mapr/dzl.log-20180524140001
-rwxrwxrwx 3 root root 1283968 2018-05-23 17:48 maprfs:///user/mapr/test
drwxr-xr-x - mapr mapr 1 2018-05-10 00:58 maprfs:///user/mapr/tmp

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Thu, 24 May 2018 15:32:40 GMT

Well, I have changed these:
Hadoop 2.x, Yarn
resource manager: localhost:8032
resource scheduler: localhost:8030

And changed my HDFS address to /user/mapr

I am still able to bring results on: "index=test"
but still getting the same error with "index=test foo"

so nothing changed 😞

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Thu, 24 May 2018 15:54:16 GMT

When I do a search for com.splunk.mr.SplunkMR on my linux system, it finds nothing. Perhaps it is missing this but Im unsure how that would be missing I figure it came with the Splunk install

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

rdagan_splunk — Thu, 24 May 2018 17:15:40 GMT

What version of Splunk are you using?
When you go to ' http://localhost:8088/conf ' you should be able to see all the correct values for the Yarn Resource Manager
yarn.resourcemanager.address and yarn.resourcemanager.scheduler.address
Can you try to run index=test1 | stats count

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Tue, 29 Sep 2020 19:37:41 GMT

Splunk 7.1
I go to that link (change out localhost for our host name where this is running) and it says it refuses to connect. ERR_CONNECTION_REFUSED

Interestingly enough, your query index=test1 | stats count does produce a result but then also gives an error:

[maproly] Error while running external process, return_code=255. See search.log for more info
[maproly] Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0002, state=FAILED, reason=Application application_1525914386605_0002 failed 2 times due to AM Container for appattempt_1525914386605_0002_000002 exited with exitCode: -1000

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

gozulin — Tue, 29 Sep 2020 19:41:20 GMT

Hi rdagan, we're using splunk 7.10.

:8088/conf doesn't work. that port is closed.

https on port 8089 works as expected but /conf is not found.

:8089/services/configs is the closest thing i could find and i don't see any Yarn stuff.

Running stats count gives us a slightly different (and shorter) error:

2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.
[maproly] Error while running external process, return_code=255. See search.log for more info
[maproly] Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0005, state=FAILED, reason=Application application_1525914386605_0005 failed 2 times due to AM Container for appattempt_1525914386605_0005_000002 exited with exitCode: -1000

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Thu, 24 May 2018 18:01:22 GMT

rdagan:
The error in our logs is:
05-24-2018 17:54:24.799 ERROR ERP.maproly - Error while invoking command: /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255

And when I try to run the class manually I get:

[root@hadoop-s1 log-gen]# /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR
Error: Could not find or load main class com.splunk.mr.SplunkMR

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

rdagan_splunk — Tue, 29 Sep 2020 19:37:47 GMT

We will need to dig into the Hadoop log - specifically the hadoop Attempt log - to see the actual error

Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0005, state=FAILED, reason=Application application_1525914386605_0005 failed 2 times due to AM Container for appattempt_1525914386605_0005_000002 exited with exitCode: -1000

Normally http://Yarn Resource Manager IP: 8088 should take you to the main Hadoop Yarn page

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

rdagan_splunk — Thu, 24 May 2018 20:39:00 GMT

Try to run this command to find which jars contain the above class. And then see if that jar is in the Hadoop classpath
find . -name "*.jar" -exec grep -Hsli com.splunk.mr.SplunkMR {} \;

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Thu, 24 May 2018 20:50:47 GMT

So I added it to the Hadoop classpath, rdagan and i see a promising result running the jar from the command line:
[root@hadoop-s1 /]# /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR
INFO SplunkMR - starting, version=6.2 ...

But when I run the search I still get the same error and even get this again...

05-24-2018 20:46:29.179 ERROR ERP.maproly - Error while invoking command: /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255

Now Im starting to wonder if it isnt some kind of permissions thing but Im running the command in CLI as root and the splunk query I think is being run by root

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Thu, 24 May 2018 21:09:35 GMT

Update:

If I run the search from the command line with:
/opt/splunk/bin/splunk search index=test1 Exiting

It seems to produce results without the error but then all of splunk is now producing no results. haha 😞
[maproly] IOException - Out of memory error while reading a very large single line input record. To skip this record set mapreduce.input.linerecordreader.line.maxlength to a lower value. Current value: 2147483647, jvm heap size: 508035072, potential value: 31752192

Working on changing the mapreduce.input.linerecordreader.line.maxlength

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Thu, 24 May 2018 21:26:41 GMT

Okay, fixed the unusual maxlength error... back to being able run the Splunk queries such as "index=test Exiting" from the CLI but not from the web UI due to the same error.

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

rdagan_splunk — Fri, 25 May 2018 05:34:02 GMT

Make sure the classpath in the Provider is the same as your Hadoop envirnment:
Specifically I am thinking about these two
vix.yarn.application.classpath
vix.mapreduce.application.classpath

https://docs.splunk.com/Documentation/Hunk/6.4.10/Hunk/RequiredConfigurationVariablesforYARN

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Fri, 25 May 2018 17:00:59 GMT

Sorry I guess I am unclear what the classpath in the Provider is...? do you mean vix.arg.1=$HADOOP_HOME/bin/hadoop

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Fri, 25 May 2018 17:31:11 GMT

I bet you mean this: /opt/mapr/hadoop/hadoop-2.7.0

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Fri, 25 May 2018 17:34:13 GMT

Well I added them to the yarn-site.xml and the error persists.

<name>vix.yarn.application.classpath</name>
<value>/opt/mapr/hadoop/hadoop-2.7.0</value>



<name>vix.mapreduce.application.classpath</name>
<value>/opt/mapr/hadoop/hadoop-2.7.0</value>

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

rdagan_splunk — Fri, 25 May 2018 20:29:45 GMT

The vix.mapreduce.. and vix.yarn must be added to the Splunk provider. At the bottom of the Provider GUI you will see the add variables option.

Re: Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79 — Fri, 25 May 2018 21:29:23 GMT

Well I added them to the variables in the Provider on Splunk UI and the error still exists.
I have submitted a ticket with Splunk Support but they dont seem to be responding. I guess maybe everyone is already in their 3 day Memorial Day Weekend. Maybe I should be too. 😛