Splunk ITSI: No events in itsi_tracked_alerts

[GowthamMagal]

Hi Splunk Gurus,

We have a splunk ITSI search head with version 4.4.3 build 14 running on Splunk version 7.2.10

I have created correlation searches.  Some of which run every minute. Event though the splunk correlation searches run as required, we have noticed that there are no events in the index itsi_tracked_alerts  created even though the alert conditions are met.

in ITSI Health check dashboard, I see this error in the internal log:

"2020-09-07 04:52:04,796 ERROR [itsi.notable_event_actions_queue_consumer] [__init__] [exception] [121502] Encountered exception when consuming. "'No key or prefix: token.'".
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-ITOA/bin/itsi_notable_event_actions_queue_consumer.py", line 109, in do_run
action_dispatch_config=action_dispatch_config
File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/event_management/itsi_notable_event_queue_consumer.py", line 130, in __init__
self.auditor = Audit(self.session_key, audit_token_name=audit_token_name)
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/notable_event_utils.py", line 553, in __init__
self.audit = PushEventManager(self.session_key, audit_token_name)
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/push_event_manager.py", line 111, in __init__
hec_token=hec_token)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/modular_input/event_writer.py", line 209, in __init__
hec_input_name, session_key, scheme, host, port, **context)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/utils.py", line 159, in wrapper
return func(*args, **kwargs)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/modular_input/event_writer.py", line 329, in _get_hec_config
return settings['port'], hec_input['token']
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/packages/splunklib/data.py", line 253, in __getitem__
raise KeyError("No key or prefix: %s" % key)
KeyError: 'No key or prefix: token.'"

And

"2020-09-07 04:52:02,514 ERROR [itsi.custom_alert.itsi_generator] [__init__] [exception] [121099] Failed to validate arguments. Please make sure arguments are correct
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-ITOA/bin/itsi_event_generator.py", line 57, in <module>
modular_alert = ItsiSendAlert(sys.stdin.read())
File "/opt/splunk/etc/apps/SA-ITOA/bin/itsi_event_generator.py", line 33, in __init__
super(ItsiSendAlert, self).__init__(settings, is_validate)
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/base_event_generation.py", line 178, in __init__
raise ValueError(_('Failed to validate arguments. Please make sure arguments are correct'))
ValueError: Failed to validate arguments. Please make sure arguments are correct"

I hope someone has faced this error and help me solve it. I have spent about 3 days looking the possible errors and going through internet resources to help me troubleshoot this

[GowthamMagal · Accepted solution]

The problem is now solved.

This ITSI instance was set up and a restore was done from another instance. There were 2 problems with the event creation in the index itsi_tracked_alerts

1) Error messages in the internal log "Encountered exception when consuming. "'No key or prefix: token.'"

2) A pop up while accessing the notable event aggregation policy

Solution:

We noticed that the HTTP tokens value concerning ITSI events were empty and proceeded with the workaround suugested in

https://docs.splunk.com/Documentation/ITSI/4.6.1/ReleaseNotes/Knownissues#Backup.2FRestore_and_Migra... #ITSI-5578

This solved both issues and now we have all the alerts and accessing Notable Event Aggregation policy works as well.

[esnyder_splunk]

Did you perform the workaround in https://docs.splunk.com/Documentation/ITSI/4.4.3/ReleaseNotes/Knownissues#Splunk_platform_issues_tha... ?

[GowthamMagal]

Hello @esnyder_splunk 

Yes, this work around is performed. But this did not solve the problem.

We happened to find the solution Just now. I will post the solution in the next post. Thank you