Splunk ITSI

Splunk can not able to break ITSI episode correctly

omersiar
Loves-to-Learn

Hello All,

We have configured our monitoring tools to have Network and Application alert events to be sent as SNMP traps.  Splunk monitors /var/log/snmp-traps.log file, parses data and indexes them, no problem there.

All necessary fields for "Correlation Search" are present (severity, title, etc), "Notable Events" are created by the ad-hoc correlation searches, searches are run for 1 minute window, also there is no problem here.

However breaking rules are not working as expected, for example there are multiple "Episodes" for same events are starting with exact same starting event, they may break prematurely and end up having more than one Episodes for the same starting event. We also observed that there are some Episodes getting just one event and never getting closed.  We have experimented with almost every combination in "Aggregation Policies"

What is going on here? Why does it get confused, I know that is hard to understand without looking actual settings and configuration but I did my best to understand documents and setting up the whole policy.  Did anyone else here had this issue?


Labels (3)
0 Karma

eduncan
Splunk Employee
Splunk Employee

You are getting multiple episodes because once you get an up or a clear, it breaks and no more NE's can enter that episode.  If you are getting a lot of episodes you may consider using a KPI instead and only create alerts when the KPI turns red.

0 Karma

eduncan
Splunk Employee
Splunk Employee

Few things to check:  Are the episodes truly duplicate (same exact number of events, same events, same timestamp)?

Check in your agg policy breaking rules - what is in there?  Make sure that it's not set to 'break for event that is normal or when flow of events is paused for - if a normal event comes in or it hits that time limit, it will break the episode and start anew.  Also is there more than one agg policy that the trap would match when it comes in?  NE's can make it into multiple episodes IF they match more than one agg policy filter.  For instance, if my filter just says:  Severity >= Normal and snmp_name =* and then I have another agg policy with a similar filter but maybe just the Severity >= Normal, the trap will match two agg policies and end up in both.  Can you post a screenshot of your breaking rules and your ACTION tab?

0 Karma

omersiar
Loves-to-Learn

Images are no longer here

0 Karma

omersiar
Loves-to-Learn

There are no entries on Action tab.

0 Karma

omersiar
Loves-to-Learn

Splunk is at 8.0.1
ITSI is at 4.4.1 Build 10

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...