Why are my Splunk IT Service Intelligence (ITSI) alert action fields incomplete for an aggregation policy action?

[mataharry]

In ITSI Aggregation policy, I set up custom actions on certain conditions. (email, or scripted alerts to a third party api ...)

When I am running action on all events of group, why has it not picked up the last or few events of the group?
Or when I have an action on a newly created group, I am missing some field like the group_ip.

It looks like the group informations are not up to date yet.

[yannK]

There are 2 possibilities here

Event had match more than one policies and it is part of two group so event state depends on which policy action had run last.
Indexing/Forwarding delay - It might be possible that we had run the action before event group information shows up in itsi_grouped_alerts.

• You can increase action_execution_delay time in $SPLUNK_HOME/etc/apps/SA-ITOA/default/itsi_rules_engine.properties.

You may have to pick a higher value, the default is 0 milliseconds. Please do not increase this time aggressively because it will impact Rules Engine performance.

see http://docs.splunk.com/Documentation/ITSI/latest/Configure/TuneNEgrouping

example, to give 1 second to wait :

action_execution_delay = 1000

The problem with this method is that it is not persistent with an ITSI upgrade (as the SA-ITOA/default will be overwritten)

• The other location were you can add a persistent delay is in the consumers inputs.conf see the default exec_delay_time = 0.1 This is the recommended method, as it will survive an upgrade. (in seconds)

look in $SPLUNK_HOME/etc/apps/SA-ITOA/default/inputs.conf for
[itsi_notable_event_actions_queue_consumer://alpha]
[itsi_notable_event_actions_queue_consumer://beta]
[itsi_notable_event_actions_queue_consumer://gamma]
[itsi_notable_event_actions_queue_consumer://zeta]
[itsi_notable_event_actions_queue_consumer://delta]

you can create in the local folder a new inputs.conf with just the stanza name and the new exec_delay_time in it.

(example to wait 1 second now)

[itsi_notable_event_actions_queue_consumer://alpha]
exec_delay_time=1

[itsi_notable_event_actions_queue_consumer://beta]
exec_delay_time=1

[itsi_notable_event_actions_queue_consumer://gamma]
exec_delay_time=1

[itsi_notable_event_actions_queue_consumer://zeta]
exec_delay_time=1

[itsi_notable_event_actions_queue_consumer://delta]
exec_delay_time=1

• There is a way to estimate the delay between the group creation and the alert action by using this method:

setup one aggregation policy that will have one action when the "group size = 1", by example an email
close your group, and wait for an event that will create a new one
use this search, to find the delay between the group being indexed, and the alert being triggered.
this is the field : delay_group_indexed-to_alert_in_milliseconds
and use it a lower boundary for your action_execution_delay

( index=itsi_grouped_alerts ) OR ( index=_internal   itsi.notable_event_actions_queue_consumer   source=*itsi_notable_event_actions_queue_consumer*.log*)
| stats  last(_indextime) AS indextime last(_time) AS ltime  by itsi_group_id index 
| eval comment="to use this search, you need to configure an action for your aggregation policy, that will trigger only one action, when the group size is exactly 1"
| eval group_creation_time=if(index="itsi_grouped_alerts", ltime,null)
| eval group_creation_time_indexed=if(index="itsi_grouped_alerts", indextime,null)
| eval alert_creation_time=if(index="_internal",ltime,null)
| stats count max(group_creation_time_indexed) AS group_creation_time_indexed max(group_creation_time) AS group_creation_time  max(alert_creation_time) AS alert_creation_time by itsi_group_id
| where count>1
| eval delay_group-alert_in_milliseconds=1000*(alert_creation_time-group_creation_time)
| eval delay_group_event_toindextime_in_milliseconds=1000*(group_creation_time_indexed-group_creation_time)
| eval delay_group_indexed-to_alert_in_milliseconds=1000*(alert_creation_time-group_creation_time_indexed)