In ITSI Aggregation policy, I set up custom actions on certain conditions. (email, or scripted alerts to a third party api ...)
When I am running action on all events of group, why has it not picked up the last or few events of the group?
Or when I have an action on a newly created group, I am missing some field like the group_ip.
It looks like the group informations are not up to date yet.
Event had match more than one policies and it is part of two group so event state depends on which policy action had run last.
Indexing/Forwarding delay - It might be possible that we had run the action before event group information shows up in itsi_grouped_alerts.
You can increase action_execution_delay time in $SPLUNK_HOME/etc/apps/SA-ITOA/default/itsi_rules_engine.properties.
You may have to pick a higher value, the default is 0 milliseconds. Please do not increase this time aggressively because it will impact Rules Engine performance.
The problem with this method is that it is not persistent with an ITSI upgrade (as the SA-ITOA/default will be overwritten)
The other location were you can add a persistent delay is in the consumers inputs.conf
see the default exec_delay_time = 0.1
This is the recommended method, as it will survive an upgrade.
look in $SPLUNK_HOME/etc/apps/SA-ITOA/default/inputs.conf for
you can create in the local folder a new inputs.conf with just the stanza name and the new exec_delay_time in it.
There is a way to estimate the delay between the group creation and the alert action by using this method:
setup one aggregation policy that will have one action when the "group size = 1", by example an email
close your group, and wait for an event that will create a new one
use this search, to find the delay between the group being indexed, and the alert being triggered.
this is the field : delay_group_indexed-to_alert_in_milliseconds
and use it a lower boundary for your action_execution_delay
( index=itsi_grouped_alerts ) OR ( index=_internal itsi.notable_event_actions_queue_consumer source=*itsi_notable_event_actions_queue_consumer*.log*)
| stats last(_indextime) AS indextime last(_time) AS ltime by itsi_group_id index
| eval comment="to use this search, you need to configure an action for your aggregation policy, that will trigger only one action, when the group size is exactly 1"
| eval group_creation_time=if(index="itsi_grouped_alerts", ltime,null)
| eval group_creation_time_indexed=if(index="itsi_grouped_alerts", indextime,null)
| eval alert_creation_time=if(index="_internal",ltime,null)
| stats count max(group_creation_time_indexed) AS group_creation_time_indexed max(group_creation_time) AS group_creation_time max(alert_creation_time) AS alert_creation_time by itsi_group_id
| where count>1
| eval delay_group-alert_in_milliseconds=1000*(alert_creation_time-group_creation_time)
| eval delay_group_event_toindextime_in_milliseconds=1000*(group_creation_time_indexed-group_creation_time)
| eval delay_group_indexed-to_alert_in_milliseconds=1000*(alert_creation_time-group_creation_time_indexed)