Hi Splunk Gurus, We have a splunk ITSI search head with version 4.4.3 build 14 running on Splunk version 7.2.10 I have created correlation searches. Some of which run every minute. Event though the splunk correlation searches run as required, we have noticed that there are no events in the index itsi_tracked_alerts created even though the alert conditions are met. in ITSI Health check dashboard, I see this error in the internal log: "2020-09-07 04:52:04,796 ERROR [itsi.notable_event_actions_queue_consumer] [__init__] [exception] [121502] Encountered exception when consuming. "'No key or prefix: token.'". Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-ITOA/bin/itsi_notable_event_actions_queue_consumer.py", line 109, in do_run action_dispatch_config=action_dispatch_config File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/event_management/itsi_notable_event_queue_consumer.py", line 130, in __init__ self.auditor = Audit(self.session_key, audit_token_name=audit_token_name) File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/notable_event_utils.py", line 553, in __init__ self.audit = PushEventManager(self.session_key, audit_token_name) File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/push_event_manager.py", line 111, in __init__ hec_token=hec_token) File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/modular_input/event_writer.py", line 209, in __init__ hec_input_name, session_key, scheme, host, port, **context) File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/utils.py", line 159, in wrapper return func(*args, **kwargs) File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/modular_input/event_writer.py", line 329, in _get_hec_config return settings['port'], hec_input['token'] File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/packages/splunklib/data.py", line 253, in __getitem__ raise KeyError("No key or prefix: %s" % key) KeyError: 'No key or prefix: token.'" And "2020-09-07 04:52:02,514 ERROR [itsi.custom_alert.itsi_generator] [__init__] [exception] [121099] Failed to validate arguments. Please make sure arguments are correct Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-ITOA/bin/itsi_event_generator.py", line 57, in <module> modular_alert = ItsiSendAlert(sys.stdin.read()) File "/opt/splunk/etc/apps/SA-ITOA/bin/itsi_event_generator.py", line 33, in __init__ super(ItsiSendAlert, self).__init__(settings, is_validate) File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/base_event_generation.py", line 178, in __init__ raise ValueError(_('Failed to validate arguments. Please make sure arguments are correct')) ValueError: Failed to validate arguments. Please make sure arguments are correct" I hope someone has faced this error and help me solve it. I have spent about 3 days looking the possible errors and going through internet resources to help me troubleshoot this
... View more