Splunk IT Service Intelligence

Splunk ITSI: No events in itsi_tracked_alerts

GowthamMagal
Explorer

Hi Splunk Gurus,

We have a splunk ITSI search head with version 4.4.3 build 14 running on Splunk version 7.2.10

I have created correlation searches.  Some of which run every minute. Event though the splunk correlation searches run as required, we have noticed that there are no events in the index itsi_tracked_alerts  created even though the alert conditions are met.

 

in ITSI Health check dashboard, I see this error in the internal log:

"2020-09-07 04:52:04,796 ERROR [itsi.notable_event_actions_queue_consumer] [__init__] [exception] [121502] Encountered exception when consuming. "'No key or prefix: token.'".
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-ITOA/bin/itsi_notable_event_actions_queue_consumer.py", line 109, in do_run
action_dispatch_config=action_dispatch_config
File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/event_management/itsi_notable_event_queue_consumer.py", line 130, in __init__
self.auditor = Audit(self.session_key, audit_token_name=audit_token_name)
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/notable_event_utils.py", line 553, in __init__
self.audit = PushEventManager(self.session_key, audit_token_name)
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/push_event_manager.py", line 111, in __init__
hec_token=hec_token)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/modular_input/event_writer.py", line 209, in __init__
hec_input_name, session_key, scheme, host, port, **context)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/utils.py", line 159, in wrapper
return func(*args, **kwargs)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/modular_input/event_writer.py", line 329, in _get_hec_config
return settings['port'], hec_input['token']
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/packages/splunklib/data.py", line 253, in __getitem__
raise KeyError("No key or prefix: %s" % key)
KeyError: 'No key or prefix: token.'"

 

And

"2020-09-07 04:52:02,514 ERROR [itsi.custom_alert.itsi_generator] [__init__] [exception] [121099] Failed to validate arguments. Please make sure arguments are correct
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-ITOA/bin/itsi_event_generator.py", line 57, in <module>
modular_alert = ItsiSendAlert(sys.stdin.read())
File "/opt/splunk/etc/apps/SA-ITOA/bin/itsi_event_generator.py", line 33, in __init__
super(ItsiSendAlert, self).__init__(settings, is_validate)
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/base_event_generation.py", line 178, in __init__
raise ValueError(_('Failed to validate arguments. Please make sure arguments are correct'))
ValueError: Failed to validate arguments. Please make sure arguments are correct"

I hope someone has faced this error and help me solve it. I have spent about 3 days looking the possible errors and going through internet resources to help me troubleshoot this

 

Labels (2)
Tags (1)
0 Karma
1 Solution

GowthamMagal
Explorer

The problem is now solved.

This ITSI instance was set up and a restore was done from another instance. There were 2 problems with the event creation in the index itsi_tracked_alerts

1) Error messages in the internal log "Encountered exception when consuming. "'No key or prefix: token.'"

2) A pop up while accessing the notable event aggregation policy

image.png

Solution:

We noticed that the HTTP tokens value concerning ITSI events were empty and proceeded with the workaround suugested in

https://docs.splunk.com/Documentation/ITSI/4.6.1/ReleaseNotes/Knownissues#Backup.2FRestore_and_Migra... #ITSI-5578

This solved both issues and now we have all the alerts and accessing Notable Event Aggregation policy works as well.

 

View solution in original post

0 Karma

GowthamMagal
Explorer

The problem is now solved.

This ITSI instance was set up and a restore was done from another instance. There were 2 problems with the event creation in the index itsi_tracked_alerts

1) Error messages in the internal log "Encountered exception when consuming. "'No key or prefix: token.'"

2) A pop up while accessing the notable event aggregation policy

image.png

Solution:

We noticed that the HTTP tokens value concerning ITSI events were empty and proceeded with the workaround suugested in

https://docs.splunk.com/Documentation/ITSI/4.6.1/ReleaseNotes/Knownissues#Backup.2FRestore_and_Migra... #ITSI-5578

This solved both issues and now we have all the alerts and accessing Notable Event Aggregation policy works as well.

 

0 Karma

esnyder_splunk
Splunk Employee
Splunk Employee
0 Karma

GowthamMagal
Explorer

Hello @esnyder_splunk 

Yes, this work around is performed. But this did not solve the problem.

We happened to find the solution Just now. I will post the solution in the next post. Thank you

 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...