In ITSI Aggregation policy, I set up custom actions on certain conditions. (email, or scripted alerts to a third party api ...)
When I am running action on all events of group, why has it not picked up the last or few events of the group?
Or when I have an action on a newly created group, I am missing some field like the group_ip.
It looks like the group informations are not up to date yet.
There are 2 possibilities here
Event had match more than one policies and it is part of two group so event state depends on which policy action had run last.
Indexing/Forwarding delay - It might be possible that we had run the action before event group information shows up in itsi_grouped_alerts.
You may have to pick a higher value, the default is 0 milliseconds. Please do not increase this time aggressively because it will impact Rules Engine performance.
example, to give 1 second to wait :
action_execution_delay = 1000
The problem with this method is that it is not persistent with an ITSI upgrade (as the SA-ITOA/default will be overwritten)
look in $SPLUNK_HOME/etc/apps/SA-ITOA/default/inputs.conf for
you can create in the local folder a new inputs.conf with just the stanza name and the new exec_delay_time in it.
(example to wait 1 second now)
[itsi_notable_event_actions_queue_consumer://alpha] exec_delay_time=1 [itsi_notable_event_actions_queue_consumer://beta] exec_delay_time=1 [itsi_notable_event_actions_queue_consumer://gamma] exec_delay_time=1 [itsi_notable_event_actions_queue_consumer://zeta] exec_delay_time=1 [itsi_notable_event_actions_queue_consumer://delta] exec_delay_time=1
setup one aggregation policy that will have one action when the "group size = 1", by example an email
close your group, and wait for an event that will create a new one
use this search, to find the delay between the group being indexed, and the alert being triggered.
this is the field : delay_group_indexed-to_alert_in_milliseconds
and use it a lower boundary for your action_execution_delay
( index=itsi_grouped_alerts ) OR ( index=_internal itsi.notable_event_actions_queue_consumer source=*itsi_notable_event_actions_queue_consumer*.log*) | stats last(_indextime) AS indextime last(_time) AS ltime by itsi_group_id index | eval comment="to use this search, you need to configure an action for your aggregation policy, that will trigger only one action, when the group size is exactly 1" | eval group_creation_time=if(index="itsi_grouped_alerts", ltime,null) | eval group_creation_time_indexed=if(index="itsi_grouped_alerts", indextime,null) | eval alert_creation_time=if(index="_internal",ltime,null) | stats count max(group_creation_time_indexed) AS group_creation_time_indexed max(group_creation_time) AS group_creation_time max(alert_creation_time) AS alert_creation_time by itsi_group_id | where count>1 | eval delay_group-alert_in_milliseconds=1000*(alert_creation_time-group_creation_time) | eval delay_group_event_toindextime_in_milliseconds=1000*(group_creation_time_indexed-group_creation_time) | eval delay_group_indexed-to_alert_in_milliseconds=1000*(alert_creation_time-group_creation_time_indexed)