Splunk IT Service Intelligence

Why are my Splunk IT Service Intelligence (ITSI) alert action fields incomplete for an aggregation policy action?

Communicator

In ITSI Aggregation policy, I set up custom actions on certain conditions. (email, or scripted alerts to a third party api ...)

When I am running action on all events of group, why has it not picked up the last or few events of the group?
Or when I have an action on a newly created group, I am missing some field like the group_ip.

It looks like the group informations are not up to date yet.

0 Karma

Splunk Employee
Splunk Employee

There are 2 possibilities here

Event had match more than one policies and it is part of two group so event state depends on which policy action had run last.
Indexing/Forwarding delay - It might be possible that we had run the action before event group information shows up in itsi_grouped_alerts.

  • You can increase action_execution_delay time in $SPLUNK_HOME/etc/apps/SA-ITOA/default/itsi_rules_engine.properties.

You may have to pick a higher value, the default is 0 milliseconds. Please do not increase this time aggressively because it will impact Rules Engine performance.

see http://docs.splunk.com/Documentation/ITSI/latest/Configure/TuneNEgrouping

example, to give 1 second to wait :

action_execution_delay = 1000

The problem with this method is that it is not persistent with an ITSI upgrade (as the SA-ITOA/default will be overwritten)

  • The other location were you can add a persistent delay is in the consumers inputs.conf see the default exec_delay_time = 0.1 This is the recommended method, as it will survive an upgrade. (in seconds)

look in $SPLUNK_HOME/etc/apps/SA-ITOA/default/inputs.conf for
[itsi_notable_event_actions_queue_consumer://alpha]
[itsi_notable_event_actions_queue_consumer://beta]
[itsi_notable_event_actions_queue_consumer://gamma]
[itsi_notable_event_actions_queue_consumer://zeta]
[itsi_notable_event_actions_queue_consumer://delta]

you can create in the local folder a new inputs.conf with just the stanza name and the new exec_delay_time in it.

(example to wait 1 second now)

[itsi_notable_event_actions_queue_consumer://alpha]
exec_delay_time=1

[itsi_notable_event_actions_queue_consumer://beta]
exec_delay_time=1

[itsi_notable_event_actions_queue_consumer://gamma]
exec_delay_time=1

[itsi_notable_event_actions_queue_consumer://zeta]
exec_delay_time=1

[itsi_notable_event_actions_queue_consumer://delta]
exec_delay_time=1
  • There is a way to estimate the delay between the group creation and the alert action by using this method:

setup one aggregation policy that will have one action when the "group size = 1", by example an email
close your group, and wait for an event that will create a new one
use this search, to find the delay between the group being indexed, and the alert being triggered.
this is the field : delay_group_indexed-to_alert_in_milliseconds
and use it a lower boundary for your action_execution_delay

( index=itsi_grouped_alerts ) OR ( index=_internal   itsi.notable_event_actions_queue_consumer   source=*itsi_notable_event_actions_queue_consumer*.log*)
| stats  last(_indextime) AS indextime last(_time) AS ltime  by itsi_group_id index 
| eval comment="to use this search, you need to configure an action for your aggregation policy, that will trigger only one action, when the group size is exactly 1"
| eval group_creation_time=if(index="itsi_grouped_alerts", ltime,null)
| eval group_creation_time_indexed=if(index="itsi_grouped_alerts", indextime,null)
| eval alert_creation_time=if(index="_internal",ltime,null)
| stats count max(group_creation_time_indexed) AS group_creation_time_indexed max(group_creation_time) AS group_creation_time  max(alert_creation_time) AS alert_creation_time by itsi_group_id
| where count>1
| eval delay_group-alert_in_milliseconds=1000*(alert_creation_time-group_creation_time)
| eval delay_group_event_toindextime_in_milliseconds=1000*(group_creation_time_indexed-group_creation_time)
| eval delay_group_indexed-to_alert_in_milliseconds=1000*(alert_creation_time-group_creation_time_indexed)
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!