Developing for Splunk Enterprise

Error invoking commnad: hadoop com.splunk.mr.SplunkMR - Return code: 255

EricLloyd79
Builder

We have the MapR filesystem and Hunk on the same node, have set up a Provider and Virtual and Im getting an error when trying to run a query other than a basic index search.

"index=test" produces correct results
"index=test keyword" produces the error below

Also see screenshot attachments of Provider and VI config.

Error from search.log:

05-23-2018 20:49:13.581 ERROR ERP.maproly - Caused by: java.lang.RuntimeException: summary_id did not exist in search info: {_tz=### SERIALIZED TIMEZONE FORMAT 1.0;C0;Y0 NW 55 54 43;$, now=1527108550.000000000, _sid=1527108550.15, site=default, _api_et=1527019200.000000000, _api_lt=1527108550.000000000, _dsi_id=0, _keySet=dghsd index::test1, _ppc.bs=$SPLUNK_ETC, _search=search index=test1 dghsd, _shp_id=C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6, _endTime=1527108550.000000000, _ppc.app=search, read_raw=1, realtime=0, _countMap=duration.command.search.expand_search;39;duration.command.search.parse_directives;0;duration.dispatch.evaluate.search;54;duration.startup.configuration;11;duration.startup.handoff;3;invocations.command.search.expand_search;1;invocations.command.search.parse_directives;1;invocations.dispatch.evaluate.search;1;invocations.startup.configuration;1;invocations.startup.handoff;1;, _ppc.user=admin, check_dangerous_command=0, _default_group=, generation_id=0, _bundle_version=0, indexed_realtime=0, search_can_be_event_type=1, indexed_realtime_offset=0, kv_store_settings=hosts;127.0.0.1:8191\;;local;127.0.0.1:8191;read_preference;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6;replica_set_name;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6;status;ready;, _timeline_events_preview=0, is_cluster_slave=0, internal_only=0, is_batch_mode=0, _remote_search=search (index=test1 dghsd) | fields keepcolorder=t "" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server", summary_stopped=0, _search_metrics={"ConsideredBuckets":0,"EliminatedBuckets":0,"ConsideredEvents":0,"TotalSlicesInBuckets":0,"DecompressedSlices":0,"FieldMetadata_Events":"","Partition":{}}, _is_summary_index=0, _search_StartUp_Spent=0, _is_keepalive=0, _is_scheduled=0, _splunkd_port=8089, _is_export=0, _is_remote=0, _maxevents=0, _search_et=1527019200.000000000, _search_lt=1527108550.000000000, _startTime=1527019200.000000000, _timestamp=1527108550.251636000, is_saved_search=0, is_remote_sorted=0, _search_StartTime=1527108550.250084000, remote_log_download_mode=disabledSavedSearches, kv_store_additional_settings=hosts_guids;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6\;;, _rt_batch_retry=0, _auth_token=8cntqHuq0Rb0Lz3T^YcThKI7mBeHBy4ki7SPCQHDHCuMQq1haa4BENOHDqd43diGvYDkRlyNuR6xs1eUwYfPE4PBO1IeTwbkxIAG2JxOpUIpE^IOBBwklXUWaqa, _drop_count=0, _provenance=UI:Search, _scan_count=0, is_shc_mode=0, rt_backfill=0, sample_seed=0, _bs_thread_count=1, _retry_count=0, _splunkd_uri=https://127.0.0.1:8089, replay_speed=0, _exported_results=0, sample_ratio=1, summary_mode=none, _query_finished=1, _optional_fields_json={}, enable_event_stream=1, _splunkd_protocol=https, _read_buckets_since_startup=0, _bs_pipeline_identifier=0, _request_finalization=0}
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.getSummaryId(SplunkMR.java:507)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.executeMapReduce(SplunkMR.java:1359)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1067)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.run(SplunkMR.java:1551)
05-23-2018 20:49:13.581 ERROR ERP.maproly - ... 3 more
05-23-2018 20:49:13.597 INFO ERP.maproly - SplunkMR - finishing, version=6.2 ...
05-23-2018 20:49:13.597 INFO ERP.maproly - DispatchReaper - Skip dispatch reaping, top level HDFS dispatch dir=/user/root/splunk/splunkmr/dispatch does not exist.
05-23-2018 20:49:13.621 ERROR ERP.maproly - Error while invoking command: /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255

alt text
alt text

Tags (3)
0 Karma
1 Solution

rdagan_splunk
Splunk Employee
Splunk Employee

Just to share the knowledge of what we found.
The error - Caused by: java.lang.RuntimeException: summary_id did not exist in search - looks like a bug in Splunk 7.1.0
Therefore, if there are features you need from 7.1 you may want to wait for Splunk to fix it.
However, if you are OK with the features of 7.0 then go to here and download 7.0.4: https://www.splunk.com/page/previous_releases#x86_64linux (you may need to login to see this page)

View solution in original post

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Just to share the knowledge of what we found.
The error - Caused by: java.lang.RuntimeException: summary_id did not exist in search - looks like a bug in Splunk 7.1.0
Therefore, if there are features you need from 7.1 you may want to wait for Splunk to fix it.
However, if you are OK with the features of 7.0 then go to here and download 7.0.4: https://www.splunk.com/page/previous_releases#x86_64linux (you may need to login to see this page)

View solution in original post

0 Karma

peterwaldispueh
New Member

Thanks for the info. A bug ID would be helpful so one can check the release notes of future splunk versions for a fix.

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

It looks as if you are using TaskTracker and not Yarn.
Change your settings to Yarn and point to Yarn resource Manager instead of Task Tracker.
You are using Hadoop 2.7, which default to Yarn.
In addition, your path to data in HDFS looks wrong. Normally all you need is /user/username

To test if you have the right location of the file in HDFS, I will recommend for you to try this command from CLI:
hadoop fs -ls maprfs:///user/mapr

0 Karma

EricLloyd79
Builder

When I do a search for com.splunk.mr.SplunkMR on my linux system, it finds nothing. Perhaps it is missing this but Im unsure how that would be missing I figure it came with the Splunk install

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

What version of Splunk are you using?
When you go to ' http://localhost:8088/conf ' you should be able to see all the correct values for the Yarn Resource Manager
yarn.resourcemanager.address and yarn.resourcemanager.scheduler.address
Can you try to run index=test1 | stats count

0 Karma

EricLloyd79
Builder

rdagan:
The error in our logs is:
05-24-2018 17:54:24.799 ERROR ERP.maproly - Error while invoking command: /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255

And when I try to run the class manually I get:

[root@hadoop-s1 log-gen]# /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR
Error: Could not find or load main class com.splunk.mr.SplunkMR

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Try to run this command to find which jars contain the above class. And then see if that jar is in the Hadoop classpath
find . -name "*.jar" -exec grep -Hsli com.splunk.mr.SplunkMR {} \;

0 Karma

EricLloyd79
Builder

So I added it to the Hadoop classpath, rdagan and i see a promising result running the jar from the command line:
[root@hadoop-s1 /]# /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR
INFO SplunkMR - starting, version=6.2 ...

But when I run the search I still get the same error and even get this again...

05-24-2018 20:46:29.179 ERROR ERP.maproly - Error while invoking command: /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255

Now Im starting to wonder if it isnt some kind of permissions thing but Im running the command in CLI as root and the splunk query I think is being run by root

0 Karma

EricLloyd79
Builder

Update:

If I run the search from the command line with:
/opt/splunk/bin/splunk search index=test1 Exiting

It seems to produce results without the error but then all of splunk is now producing no results. haha 😞
[maproly] IOException - Out of memory error while reading a very large single line input record. To skip this record set mapreduce.input.linerecordreader.line.maxlength to a lower value. Current value: 2147483647, jvm heap size: 508035072, potential value: 31752192

Working on changing the mapreduce.input.linerecordreader.line.maxlength

0 Karma

EricLloyd79
Builder

Okay, fixed the unusual maxlength error... back to being able run the Splunk queries such as "index=test Exiting" from the CLI but not from the web UI due to the same error.

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Make sure the classpath in the Provider is the same as your Hadoop envirnment:
Specifically I am thinking about these two
vix.yarn.application.classpath
vix.mapreduce.application.classpath

https://docs.splunk.com/Documentation/Hunk/6.4.10/Hunk/RequiredConfigurationVariablesforYARN

0 Karma

EricLloyd79
Builder

Sorry I guess I am unclear what the classpath in the Provider is...? do you mean vix.arg.1=$HADOOP_HOME/bin/hadoop

0 Karma

EricLloyd79
Builder

I bet you mean this: /opt/mapr/hadoop/hadoop-2.7.0

0 Karma

EricLloyd79
Builder

Well I added them to the yarn-site.xml and the error persists.

<name>vix.yarn.application.classpath</name>
<value>/opt/mapr/hadoop/hadoop-2.7.0</value>



<name>vix.mapreduce.application.classpath</name>
<value>/opt/mapr/hadoop/hadoop-2.7.0</value>
0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

The vix.mapreduce.. and vix.yarn must be added to the Splunk provider. At the bottom of the Provider GUI you will see the add variables option.

0 Karma

EricLloyd79
Builder

Well I added them to the variables in the Provider on Splunk UI and the error still exists.
I have submitted a ticket with Splunk Support but they dont seem to be responding. I guess maybe everyone is already in their 3 day Memorial Day Weekend. Maybe I should be too. 😛

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

It looks like you are very close to fixing this classpath issue.
Earlier when you fixed the Hadoop ClassPath, Have you modified the Hadoop classpath permanently in the env.sh file, and restarted Hadoop?

If I am not mistaken Hadoop classpath is set in /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/env.sh

Basically what I am asking, is there a way for you to run Splunk using the same classpath you used to fix your Hadoop?

0 Karma

EricLloyd79
Builder

We are working with some guys at MapR to get this resolved but now even Splunk queries run dont show up in the resource manager.
We used a basic mapreduce test script for yarn to see if it shows up and it does so theres now a disconnect between the mapreduce yarn resource manager and Splunk. I used the resource manager address from MapR UI as a parameter for the Provider.
I am unsure how to restart hadoop itself and I don think the Hadoop classpath was set permanently... Im not sure what your last sentence entails running Splunk using the same classpath as used to fix Hadoop. Im hesitant to change anything now that the guys at MapR are working on a solution but am trying different configs with a new 'test' provider.

0 Karma

EricLloyd79
Builder

well it seems we got in back in ResourceManger ... we were running in Verbose Mode and just running an index=xyz query.

We see it accepted and then it fails with this error now:
[maproly] Error while running external process, return_code=255. See search.log for more info
[maproly] Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1527615556973_0004, state=FAILED, reason=Application application_1527615556973_0004 failed 2 times due to AM Container for appattempt_1527615556973_0004_000002 exited with exitCode: 1

0 Karma

EricLloyd79
Builder

Hm, so that error was appearing due to setting the
vix.yarn.application.classpath
vix.mapreduce.application.classpath
manually in the Provider UI to
/opt/mapr/hadoop/hadoop-2.7.0

Once I created a new Provider that is identical but without the custom settings above it went into MapReduce ResourceManager fine and finished.
And now Im back to my original error 😞

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!