Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

[EricLloyd79]

We have the MapR filesystem and Hunk on the same node, have set up a Provider and Virtual and Im getting an error when trying to run a query other than a basic index search.

"index=test" produces correct results
"index=test keyword" produces the error below

Also see screenshot attachments of Provider and VI config.

Error from search.log:

05-23-2018 20:49:13.581 ERROR ERP.maproly - Caused by: java.lang.RuntimeException: summary_id did not exist in search info: {_tz=### SERIALIZED TIMEZONE FORMAT 1.0;C0;Y0 NW 55 54 43;$, now=1527108550.000000000, _sid=1527108550.15, site=default, _api_et=1527019200.000000000, _api_lt=1527108550.000000000, _dsi_id=0, _keySet=dghsd index::test1, _ppc.bs=$SPLUNK_ETC, _search=search index=test1 dghsd, _shp_id=C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6, _endTime=1527108550.000000000, _ppc.app=search, read_raw=1, realtime=0, _countMap=duration.command.search.expand_search;39;duration.command.search.parse_directives;0;duration.dispatch.evaluate.search;54;duration.startup.configuration;11;duration.startup.handoff;3;invocations.command.search.expand_search;1;invocations.command.search.parse_directives;1;invocations.dispatch.evaluate.search;1;invocations.startup.configuration;1;invocations.startup.handoff;1;, _ppc.user=admin, check_dangerous_command=0, _default_group=, generation_id=0, _bundle_version=0, indexed_realtime=0, search_can_be_event_type=1, indexed_realtime_offset=0, kv_store_settings=hosts;127.0.0.1:8191\;;local;127.0.0.1:8191;read_preference;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6;replica_set_name;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6;status;ready;, _timeline_events_preview=0, is_cluster_slave=0, internal_only=0, is_batch_mode=0, _remote_search=search (index=test1 dghsd) | fields keepcolorder=t "" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server", summary_stopped=0, _search_metrics={"ConsideredBuckets":0,"EliminatedBuckets":0,"ConsideredEvents":0,"TotalSlicesInBuckets":0,"DecompressedSlices":0,"FieldMetadata_Events":"","Partition":{}}, _is_summary_index=0, _search_StartUp_Spent=0, _is_keepalive=0, _is_scheduled=0, _splunkd_port=8089, _is_export=0, _is_remote=0, _maxevents=0, _search_et=1527019200.000000000, _search_lt=1527108550.000000000, _startTime=1527019200.000000000, _timestamp=1527108550.251636000, is_saved_search=0, is_remote_sorted=0, _search_StartTime=1527108550.250084000, remote_log_download_mode=disabledSavedSearches, kv_store_additional_settings=hosts_guids;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6\;;, _rt_batch_retry=0, _auth_token=8cntqHuq0Rb0Lz3T^YcThKI7mBeHBy4ki7SPCQHDHCuMQq1haa4BENOHDqd43diGvYDkRlyNuR6xs1eUwYfPE4PBO1IeTwbkxIAG2JxOpUIpE^IOBBwklXUWaqa, _drop_count=0, _provenance=UI:Search, _scan_count=0, is_shc_mode=0, rt_backfill=0, sample_seed=0, _bs_thread_count=1, _retry_count=0, _splunkd_uri=https://127.0.0.1:8089, replay_speed=0, _exported_results=0, sample_ratio=1, summary_mode=none, _query_finished=1, _optional_fields_json={}, enable_event_stream=1, _splunkd_protocol=https, _read_buckets_since_startup=0, _bs_pipeline_identifier=0, _request_finalization=0}
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.getSummaryId(SplunkMR.java:507)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.executeMapReduce(SplunkMR.java:1359)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1067)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.run(SplunkMR.java:1551)
05-23-2018 20:49:13.581 ERROR ERP.maproly - ... 3 more
05-23-2018 20:49:13.597 INFO ERP.maproly - SplunkMR - finishing, version=6.2 ...
05-23-2018 20:49:13.597 INFO ERP.maproly - DispatchReaper - Skip dispatch reaping, top level HDFS dispatch dir=/user/root/splunk/splunkmr/dispatch does not exist.
05-23-2018 20:49:13.621 ERROR ERP.maproly - Error while invoking command: /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255

[gozulin]

Hi rdagan, we're using splunk 7.10.

:8088/conf doesn't work. that port is closed.

https on port 8089 works as expected but /conf is not found.

:8089/services/configs is the closest thing i could find and i don't see any Yarn stuff.

Running stats count gives us a slightly different (and shorter) error:

2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.
[maproly] Error while running external process, return_code=255. See search.log for more info
[maproly] Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0005, state=FAILED, reason=Application application_1525914386605_0005 failed 2 times due to AM Container for appattempt_1525914386605_0005_000002 exited with exitCode: -1000

[rdagan_splunk]

We will need to dig into the Hadoop log - specifically the hadoop Attempt log - to see the actual error

Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0005, state=FAILED, reason=Application application_1525914386605_0005 failed 2 times due to AM Container for appattempt_1525914386605_0005_000002 exited with exitCode: -1000

Normally http://Yarn Resource Manager IP: 8088 should take you to the main Hadoop Yarn page

[EricLloyd79]

Splunk 7.1
I go to that link (change out localhost for our host name where this is running) and it says it refuses to connect. ERR_CONNECTION_REFUSED

Interestingly enough, your query index=test1 | stats count does produce a result but then also gives an error:

[maproly] Error while running external process, return_code=255. See search.log for more info
[maproly] Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0002, state=FAILED, reason=Application application_1525914386605_0002 failed 2 times due to AM Container for appattempt_1525914386605_0002_000002 exited with exitCode: -1000

[EricLloyd79]

Well, I have changed these:
Hadoop 2.x, Yarn
resource manager: localhost:8032
resource scheduler: localhost:8030

And changed my HDFS address to /user/mapr

I am still able to bring results on: "index=test"
but still getting the same error with "index=test foo"

so nothing changed 😞

[EricLloyd79]

Thank you for your reply. Do you know how I can discover the path to my Yarn Resource Manager?

Its interesting you say that the location of the file in HDFS is wrong because that is definitely where it is. I am also able to bring up that data via that directory with a simple query of "index=test" so it is finding it.
When I run your command I get:

[root@hadoop-s1 elloyd]# hadoop fs -ls maprfs:///user/mapr
Found 22 items
drwxr-xr-x - root root 1 2018-05-23 21:57 maprfs:///user/mapr/2018
drwxr-xr-x - mapr mapr 1 2018-05-10 00:56 maprfs:///user/mapr/drill
-rw-r--r-- 3 root root 0 2018-05-24 14:00 maprfs:///user/mapr/dzl.log
-rw-r--r-- 3 root root 0 2018-05-23 22:00 maprfs:///user/mapr/dzl.log-20180523220001
-rw-r--r-- 3 root root 0 2018-05-23 23:00 maprfs:///user/mapr/dzl.log-20180523230001
-rw-r--r-- 3 root root 0 2018-05-24 00:00 maprfs:///user/mapr/dzl.log-20180524000001
-rw-r--r-- 3 root root 0 2018-05-24 01:00 maprfs:///user/mapr/dzl.log-20180524010002
-rw-r--r-- 3 root root 0 2018-05-24 02:00 maprfs:///user/mapr/dzl.log-20180524020001
-rw-r--r-- 3 root root 0 2018-05-24 03:00 maprfs:///user/mapr/dzl.log-20180524030001
-rw-r--r-- 3 root root 0 2018-05-24 04:00 maprfs:///user/mapr/dzl.log-20180524040001
-rw-r--r-- 3 root root 0 2018-05-24 05:00 maprfs:///user/mapr/dzl.log-20180524050002
-rw-r--r-- 3 root root 0 2018-05-24 06:00 maprfs:///user/mapr/dzl.log-20180524060002
-rw-r--r-- 3 root root 0 2018-05-24 07:00 maprfs:///user/mapr/dzl.log-20180524070001
-rw-r--r-- 3 root root 0 2018-05-24 08:00 maprfs:///user/mapr/dzl.log-20180524080001
-rw-r--r-- 3 root root 0 2018-05-24 09:00 maprfs:///user/mapr/dzl.log-20180524090001
-rw-r--r-- 3 root root 0 2018-05-24 10:00 maprfs:///user/mapr/dzl.log-20180524100001
-rw-r--r-- 3 root root 0 2018-05-24 11:00 maprfs:///user/mapr/dzl.log-20180524110002
-rw-r--r-- 3 root root 0 2018-05-24 12:00 maprfs:///user/mapr/dzl.log-20180524120001
-rw-r--r-- 3 root root 0 2018-05-24 13:00 maprfs:///user/mapr/dzl.log-20180524130001
-rw-r--r-- 3 root root 0 2018-05-24 14:00 maprfs:///user/mapr/dzl.log-20180524140001
-rwxrwxrwx 3 root root 1283968 2018-05-23 17:48 maprfs:///user/mapr/test
drwxr-xr-x - mapr mapr 1 2018-05-10 00:58 maprfs:///user/mapr/tmp