We have the MapR filesystem and Hunk on the same node, have set up a Provider and Virtual and Im getting an error when trying to run a query other than a basic index search.
"index=test" produces correct results
"index=test keyword" produces the error below
Also see screenshot attachments of Provider and VI config.
Error from search.log:
05-23-2018 20:49:13.581 ERROR ERP.maproly - Caused by: java.lang.RuntimeException: summary_id did not exist in search info: {_tz=### SERIALIZED TIMEZONE FORMAT 1.0;C0;Y0 NW 55 54 43;$, now=1527108550.000000000, _sid=1527108550.15, site=default, _api_et=1527019200.000000000, _api_lt=1527108550.000000000, _dsi_id=0, _keySet=dghsd index::test1, _ppc.bs=$SPLUNK_ETC, _search=search index=test1 dghsd, _shp_id=C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6, _endTime=1527108550.000000000, _ppc.app=search, read_raw=1, realtime=0, _countMap=duration.command.search.expand_search;39;duration.command.search.parse_directives;0;duration.dispatch.evaluate.search;54;duration.startup.configuration;11;duration.startup.handoff;3;invocations.command.search.expand_search;1;invocations.command.search.parse_directives;1;invocations.dispatch.evaluate.search;1;invocations.startup.configuration;1;invocations.startup.handoff;1;, _ppc.user=admin, check_dangerous_command=0, _default_group=, generation_id=0, _bundle_version=0, indexed_realtime=0, search_can_be_event_type=1, indexed_realtime_offset=0, kv_store_settings=hosts;127.0.0.1:8191\;;local;127.0.0.1:8191;read_preference;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6;replica_set_name;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6;status;ready;, _timeline_events_preview=0, is_cluster_slave=0, internal_only=0, is_batch_mode=0, _remote_search=search (index=test1 dghsd) | fields keepcolorder=t "" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server", summary_stopped=0, _search_metrics={"ConsideredBuckets":0,"EliminatedBuckets":0,"ConsideredEvents":0,"TotalSlicesInBuckets":0,"DecompressedSlices":0,"FieldMetadata_Events":"","Partition":{}}, _is_summary_index=0, _search_StartUp_Spent=0, _is_keepalive=0, _is_scheduled=0, _splunkd_port=8089, _is_export=0, _is_remote=0, _maxevents=0, _search_et=1527019200.000000000, _search_lt=1527108550.000000000, _startTime=1527019200.000000000, _timestamp=1527108550.251636000, is_saved_search=0, is_remote_sorted=0, _search_StartTime=1527108550.250084000, remote_log_download_mode=disabledSavedSearches, kv_store_additional_settings=hosts_guids;C0F05B71-38F6-4B2F-ACF7-5756DCD4CAB6\;;, _rt_batch_retry=0, _auth_token=8cntqHuq0Rb0Lz3T^YcThKI7mBeHBy4ki7SPCQHDHCuMQq1haa4BENOHDqd43diGvYDkRlyNuR6xs1eUwYfPE4PBO1IeTwbkxIAG2JxOpUIpE^IOBBwklXUWaqa, _drop_count=0, _provenance=UI:Search, _scan_count=0, is_shc_mode=0, rt_backfill=0, sample_seed=0, _bs_thread_count=1, _retry_count=0, _splunkd_uri=https://127.0.0.1:8089, replay_speed=0, _exported_results=0, sample_ratio=1, summary_mode=none, _query_finished=1, _optional_fields_json={}, enable_event_stream=1, _splunkd_protocol=https, _read_buckets_since_startup=0, _bs_pipeline_identifier=0, _request_finalization=0}
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.getSummaryId(SplunkMR.java:507)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.executeMapReduce(SplunkMR.java:1359)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1067)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802)
05-23-2018 20:49:13.581 ERROR ERP.maproly - at com.splunk.mr.SplunkMR.run(SplunkMR.java:1551)
05-23-2018 20:49:13.581 ERROR ERP.maproly - ... 3 more
05-23-2018 20:49:13.597 INFO ERP.maproly - SplunkMR - finishing, version=6.2 ...
05-23-2018 20:49:13.597 INFO ERP.maproly - DispatchReaper - Skip dispatch reaping, top level HDFS dispatch dir=/user/root/splunk/splunkmr/dispatch does not exist.
05-23-2018 20:49:13.621 ERROR ERP.maproly - Error while invoking command: /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255
Just to share the knowledge of what we found.
The error - Caused by: java.lang.RuntimeException: summary_id did not exist in search - looks like a bug in Splunk 7.1.0
Therefore, if there are features you need from 7.1 you may want to wait for Splunk to fix it.
However, if you are OK with the features of 7.0 then go to here and download 7.0.4: https://www.splunk.com/page/previous_releases#x86_64linux (you may need to login to see this page)
Hi rdagan, we're using splunk 7.10.
:8088/conf doesn't work. that port is closed.
https on port 8089 works as expected but /conf is not found.
:8089/services/configs is the closest thing i could find and i don't see any Yarn stuff.
Running stats count gives us a slightly different (and shorter) error:
2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.
[maproly] Error while running external process, return_code=255. See search.log for more info
[maproly] Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0005, state=FAILED, reason=Application application_1525914386605_0005 failed 2 times due to AM Container for appattempt_1525914386605_0005_000002 exited with exitCode: -1000
We will need to dig into the Hadoop log - specifically the hadoop Attempt log - to see the actual error
Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0005, state=FAILED, reason=Application application_1525914386605_0005 failed 2 times due to AM Container for appattempt_1525914386605_0005_000002 exited with exitCode: -1000
Normally http://Yarn Resource Manager IP: 8088 should take you to the main Hadoop Yarn page
Splunk 7.1
I go to that link (change out localhost for our host name where this is running) and it says it refuses to connect. ERR_CONNECTION_REFUSED
Interestingly enough, your query index=test1 | stats count does produce a result but then also gives an error:
[maproly] Error while running external process, return_code=255. See search.log for more info
[maproly] Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0002, state=FAILED, reason=Application application_1525914386605_0002 failed 2 times due to AM Container for appattempt_1525914386605_0002_000002 exited with exitCode: -1000
Well, I have changed these:
Hadoop 2.x, Yarn
resource manager: localhost:8032
resource scheduler: localhost:8030
And changed my HDFS address to /user/mapr
I am still able to bring results on: "index=test"
but still getting the same error with "index=test foo"
so nothing changed 😞
Thank you for your reply. Do you know how I can discover the path to my Yarn Resource Manager?
Its interesting you say that the location of the file in HDFS is wrong because that is definitely where it is. I am also able to bring up that data via that directory with a simple query of "index=test" so it is finding it.
When I run your command I get:
[root@hadoop-s1 elloyd]# hadoop fs -ls maprfs:///user/mapr
Found 22 items
drwxr-xr-x - root root 1 2018-05-23 21:57 maprfs:///user/mapr/2018
drwxr-xr-x - mapr mapr 1 2018-05-10 00:56 maprfs:///user/mapr/drill
-rw-r--r-- 3 root root 0 2018-05-24 14:00 maprfs:///user/mapr/dzl.log
-rw-r--r-- 3 root root 0 2018-05-23 22:00 maprfs:///user/mapr/dzl.log-20180523220001
-rw-r--r-- 3 root root 0 2018-05-23 23:00 maprfs:///user/mapr/dzl.log-20180523230001
-rw-r--r-- 3 root root 0 2018-05-24 00:00 maprfs:///user/mapr/dzl.log-20180524000001
-rw-r--r-- 3 root root 0 2018-05-24 01:00 maprfs:///user/mapr/dzl.log-20180524010002
-rw-r--r-- 3 root root 0 2018-05-24 02:00 maprfs:///user/mapr/dzl.log-20180524020001
-rw-r--r-- 3 root root 0 2018-05-24 03:00 maprfs:///user/mapr/dzl.log-20180524030001
-rw-r--r-- 3 root root 0 2018-05-24 04:00 maprfs:///user/mapr/dzl.log-20180524040001
-rw-r--r-- 3 root root 0 2018-05-24 05:00 maprfs:///user/mapr/dzl.log-20180524050002
-rw-r--r-- 3 root root 0 2018-05-24 06:00 maprfs:///user/mapr/dzl.log-20180524060002
-rw-r--r-- 3 root root 0 2018-05-24 07:00 maprfs:///user/mapr/dzl.log-20180524070001
-rw-r--r-- 3 root root 0 2018-05-24 08:00 maprfs:///user/mapr/dzl.log-20180524080001
-rw-r--r-- 3 root root 0 2018-05-24 09:00 maprfs:///user/mapr/dzl.log-20180524090001
-rw-r--r-- 3 root root 0 2018-05-24 10:00 maprfs:///user/mapr/dzl.log-20180524100001
-rw-r--r-- 3 root root 0 2018-05-24 11:00 maprfs:///user/mapr/dzl.log-20180524110002
-rw-r--r-- 3 root root 0 2018-05-24 12:00 maprfs:///user/mapr/dzl.log-20180524120001
-rw-r--r-- 3 root root 0 2018-05-24 13:00 maprfs:///user/mapr/dzl.log-20180524130001
-rw-r--r-- 3 root root 0 2018-05-24 14:00 maprfs:///user/mapr/dzl.log-20180524140001
-rwxrwxrwx 3 root root 1283968 2018-05-23 17:48 maprfs:///user/mapr/test
drwxr-xr-x - mapr mapr 1 2018-05-10 00:58 maprfs:///user/mapr/tmp