Error while search data from Hadoop

[arcdevil]

Hello!

I try Splunk Analytics on Hadoop in the test zone.

Configured the provider, configured the virtual index, with a simple search (index = hadoop_test) the result is returned and everything is fine.

But when I add additional conditions, for example source / sourcetype always returns the following error.

2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.   

    [5_13] Error while running external process, return_code=255. See search.log for more info
    [5_13] Exception - java.lang.RuntimeException: summary_id did not exist in search info: {_tz=### SERIALIZED TIMEZONE FORMAT 1.0;Y9017 NW 4C 4D 54;Y9017 NW 4D 4D 54;Y12679 YW 4D 53 54;Y9079 NW 4D 4D 54;Y16279 YW 4D 44 53 54;Y14400 YG 4D 53 44;Y10800 NW 4D 53 4B;Y14400 YW 4D 53 44;Y18000 YW 2B 30 35;Y7200 NW 45 45 54;Y10800 NS 4D 53 4B;Y14400 YS 4D 53 44;Y10800 YS 45 45 53 54;Y7200 NS 45 45 54;Y14400 NS 4D 53 4B;Y14400 YS 4D 53 44;Y10800 NS 4D 53 4B;@-2840149817 1;@-1688265017 3;@-1656819079 2;@-1641353479 3;@-1627965079 4;@-1618716679 2;@-1596429079 4;@-1593820800 5;@-1589860800 6;@-1542427200 7;@-1539493200 8;@-1525323600 7;@-1522728000 6;@-1491188400 9;@-1247536800 6;@354920400 7;@370728000 6;@386456400 7;@402264000 6;@417992400 7;@433800000 6;@449614800 7;@465346800 10;@481071600 11;@496796400 10;@512521200 11;@528246000 10;@543970800 11;@559695600 10;@575420400 11;@591145200 10;@606870000 11;@622594800 10;@638319600 11;@654649200 10;@670374000 12;@686102400 13;@695779200 10;@701823600 11;@717548400 10;@733273200 11;@748998000 10;@764722800 11;@780447600 10;@796172400 11;@811897200 10;@828226800 11;@846370800 10;@859676400 11;@877820400 10;@891126000 11;@909270000 10;@922575600 11;@941324400 10;@954025200 11;@972774000 10;@985474800 11;@1004223600 10;@1017529200 11;@1035673200 10;@1048978800 11;@1067122800 10;@1080428400 11;@1099177200 10;@1111878000 11;@1130626800 10;@1143327600 11;@1162076400 10;@1174777200 11;@1193526000 10;@1206831600 11;@1224975600 10;@1238281200 11;@1256425200 10;@1269730800 11;@1288479600 10;@1301180400 14;@1414274400 10;$, now=1526642535.000000000, _sid=1526642535.12, site=default, _api_et=1526554800.000000000, _api_lt=1526642535.000000000, _dsi_id=0, _keySet=index::hadoop_test source::/user/splunk/anaconda.storage.log, _ppc.bs=$SPLUNK_ETC, _search=search index=hadoop_test source="/user/splunk/anaconda.storage.log", _shp_id=11259D3D-008A-4FB8-A329-A38D5B1D948A, _endTime=1526642535.000000000, _ppc.app=search, read_raw=1, realtime=0, _countMap=duration.command.search.expand_search;57;duration.command.search.parse_directives;0;duration.dispatch.evaluate.search;68;duration.startup.configuration;9;duration.startup.handoff;43;invocations.command.search.expand_search;1;invocations.command.search.parse_directives;1;invocations.dispatch.evaluate.search;1;invocations.startup.configuration;1;invocations.startup.handoff;1;, _ppc.user=admin, check_dangerous_command=1, generation_id=0, _bundle_version=0, indexed_realtime=0, search_can_be_event_type=1, indexed_realtime_offset=0, kv_store_settings=hosts;127.0.0.1:8191\;;local;127.0.0.1:8191;read_preference;11259D3D-008A-4FB8-A329-A38D5B1D948A;replica_set_name;11259D3D-008A-4FB8-A329-A38D5B1D948A;status;ready;, _timeline_events_preview=0, is_cluster_slave=0, internal_only=0, is_batch_mode=0, _remote_search=search (index=hadoop_test source="/user/splunk/anaconda.storage.log") | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server", summary_stopped=0, _search_metrics={"ConsideredBuckets":0,"EliminatedBuckets":0,"ConsideredEvents":0,"TotalSlicesInBuckets":0,"DecompressedSlices":0,"FieldMetadata_Events":"","Partition":{}}, _is_summary_index=0, _search_StartUp_Spent=0, _is_keepalive=0, _is_scheduled=0, _splunkd_port=8089, _is_export=0, _is_remote=0, _maxevents=0, _search_et=1526554800.000000000, _search_lt=1526642535.000000000, _startTime=1526554800.000000000, _timestamp=1526642535.172861000, is_saved_search=0, is_remote_sorted=0, _search_StartTime=1526642535.172061000, remote_log_download_mode=disabledSavedSearches, kv_store_additional_settings=hosts_guids;11259D3D-008A-4FB8-A329-A38D5B1D948A\;;, _rt_batch_retry=0, _auth_token=BDOelxDhBhVAs2afGYwCyerDTllb3LxtQFDtYTsESoRbSmfIDrM90g5OfDA8AWFX1lf0la5ejNDf59RIlvzTWkY3fGSaSx3gi_8xF^20lo7Qlhi^^Ug4yoWMBAdo, _drop_count=0, _provenance=UI:Search, _scan_count=0, is_shc_mode=0, rt_backfill=0, sample_seed=0, _bs_thread_count=1, _retry_count=0, _splunkd_uri=https://127.0.0.1:8089, replay_speed=0, _exported_results=0, sample_ratio=1, summary_mode=none, _query_finished=1, _optional_fields_json={}, enable_event_stream=1, _splunkd_protocol=https, _read_buckets_since_startup=0, _bs_pipeline_identifier=0, _request_finalization=0}

Search.log shows the same error, just like the Java error stack.

Can anyone tell me how to solve this problem? 🙂

[peterwaldispueh]

Thanks for the info. A bug ID would be helpful so one can check the release notes of future splunk versions for a fix.

[rdagan_splunk]

After running some tests we can see that the error - Caused by: java.lang.RuntimeException: summary_id did not exist in search - looks like a bug in Splunk 7.1.0
Therefore, if there are features you need from 7.1 you may want to wait for Splunk to fix it.
However, if you are OK with the features of 7.0 then go to here and download 7.0.4: https://www.splunk.com/page/previous_releases#x86_64linux (you may need to login to see this page)

[rdagan_splunk]

Can you enable the Debug mode in the provider and see if you get any additional information?

For example, do you see any errors that shows failed Hadoop MR tasks or attempts?

[arcdevil]

search.log - https://dropmefiles.com/660047

[gozulin]

We're working with mapr to identify the problem. They did their own install and had the same issue so at least it's easy to reproduce.

[rdagan_splunk]

This error normally indicates that your Yarn Resource Manager and Yarn Resource Manager Scheduler IP and Port are wrong.

[arcdevil]

cloudera yarn configuration - https://ibb.co/f54KD8
splunk provider configuration - https://ibb.co/bxOCY8
cloudera iptables - https://ibb.co/iKkgRT

[gozulin]

I am having the exact same issue. Using splunk 7 and mapr 5.22

[arcdevil]

Splunk 7.1 and Cloudera 5.13