Splunk ITSI

Splunk can not able to break ITSI episode correctly

omersiar
Loves-to-Learn

Hello All,

We have configured our monitoring tools to have Network and Application alert events to be sent as SNMP traps.  Splunk monitors /var/log/snmp-traps.log file, parses data and indexes them, no problem there.

All necessary fields for "Correlation Search" are present (severity, title, etc), "Notable Events" are created by the ad-hoc correlation searches, searches are run for 1 minute window, also there is no problem here.

However breaking rules are not working as expected, for example there are multiple "Episodes" for same events are starting with exact same starting event, they may break prematurely and end up having more than one Episodes for the same starting event. We also observed that there are some Episodes getting just one event and never getting closed.  We have experimented with almost every combination in "Aggregation Policies"

What is going on here? Why does it get confused, I know that is hard to understand without looking actual settings and configuration but I did my best to understand documents and setting up the whole policy.  Did anyone else here had this issue?


Labels (3)
0 Karma

eduncan
Splunk Employee
Splunk Employee

You are getting multiple episodes because once you get an up or a clear, it breaks and no more NE's can enter that episode.  If you are getting a lot of episodes you may consider using a KPI instead and only create alerts when the KPI turns red.

0 Karma

eduncan
Splunk Employee
Splunk Employee

Few things to check:  Are the episodes truly duplicate (same exact number of events, same events, same timestamp)?

Check in your agg policy breaking rules - what is in there?  Make sure that it's not set to 'break for event that is normal or when flow of events is paused for - if a normal event comes in or it hits that time limit, it will break the episode and start anew.  Also is there more than one agg policy that the trap would match when it comes in?  NE's can make it into multiple episodes IF they match more than one agg policy filter.  For instance, if my filter just says:  Severity >= Normal and snmp_name =* and then I have another agg policy with a similar filter but maybe just the Severity >= Normal, the trap will match two agg policies and end up in both.  Can you post a screenshot of your breaking rules and your ACTION tab?

0 Karma

omersiar
Loves-to-Learn

Images are no longer here

0 Karma

omersiar
Loves-to-Learn

There are no entries on Action tab.

0 Karma

omersiar
Loves-to-Learn

Splunk is at 8.0.1
ITSI is at 4.4.1 Build 10

0 Karma
Get Updates on the Splunk Community!

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...

Splunk With AppDynamics - Meet the New IT (And Engineering) Couple

Wednesday, November 20, 2024  |  10AM PT / 1PM ET Register Now Join us in this session to learn all about ...