topic Duration between two events with conditions in Splunk Search

Duration between two events with conditions

pgraf — Tue, 06 Jul 2021 07:40:47 GMT

Hi guys

Im pretty new to Splunk and do not know how to create the search I need.

We are forwarding events from our Faul Monitoring toward Splunk. there are three Type of event having the same fields. The three types are SET, UPDATE and CLEAR. So if an alarm is raised SET is the first event in Splunk afterward if more fields are filled in the monitoring UPDATE events are coming to splunk. Depending on the fault several (n) update events can be there. If the fault is closed the CLEAR event is received by Splunk.

At some point the field "TTID" in an update event contains a TTID but all following UPDATES will contain it as well. What I try to achiev is searching for SET AND UPDATE and calculate the duration between the SET and the first UPDATE which is containing *INC* in the TTID field

I've created a search to get duration between SET and CLEAR but as UPDATE can be there n-times I do not know how to really get the time between the first UPDATE containing *INC* in the TTID field

Thanks a lot for your help

Re: Duration between two events with conditions

ITWhisperer — Tue, 06 Jul 2021 08:09:47 GMT

Can you provide some (anonymised) sample events so we can see what you are dealing with. Also, what does your current search look like? Please provide in code blocks </> so it is easier for those of us with poor eyesight to read, and to use as test data for our proposed solutions.

By the way, how do you know which CLEAR relates to which SET if the TTID isn't provided until an UPDATE?

Re: Duration between two events with conditions

pgraf — Tue, 06 Jul 2021 08:40:41 GMT

I've anonymised an SET and UPDATE event which you can see below. To find the related events I'm using the field "AlarmID".

Unfortunately, I'm not really having an idea how to properly do it, so I don't have any search for that particular request but I'll past the search I'm using to get duration till clear.

Thanks a lot

Event Example

This is the UPDATE Event
============================================================
Current Date and Time is : Mon Jul 5 17:47:31 CEST 2021
Alarm Notification from XXXXXXXXXX

Alarm UPDATEd:
UpdateDate: 07/05/2021
UpdateTime: 17:47:31
DeviceType: XXXXXXXXXXX
Mtype: XXXXXXXX
ModelName: XXXXXXXXXX
AlarmID: 6755882
AlarmTitle: Issue with Network Device
TTID: INCXXXXXXXX
GlobalAlarmID: 12345678-12345678-12344566
Severity: CRITICAL
ProbableCauseID: 10009
RepairPerson:
AlarmStatus: XXXXXXXXX
IPAddress: xx.xx.xx.xx
AlarmState: NEW
Acknowledged: FALSE
UserClearable: FALSE
Location:
AlarmAge: 0
NotificationData:
ProbableCause: Issue with Network Device
IfName: XXXXXXXXX
IfDesc: XXXXXXXXX
IfAlias: XXXXXXXXXX
TicketStatus: XXXXXXXXXX
CRQ_ID:
Maintenance:
EnrichmentInfo: Processing_time_00:06:04
============================================================

This is the SET event
============================================================
Current Date and Time is : Mon Jul 5 17:38:28 CEST 2021
Alarm Notification from XXXXXXXX

Alarm SET:
SetDate: 07/05/2021
SetTime: 17:38:27
DeviceType: XXXXXXXXXXX
Mtype: XXXXXXXXXXX
ModelName: XXXXXXXXXXX
AlarmID: 6755882
AlarmTitle: Issue with Network Device
TTID:
GlobalAlarmID: 12345678-12345678-12344566
Severity: CRITICAL
ProbableCauseID: 10009
RepairPerson:
AlarmStatus:
IPAddress: XXXXXXXXXXX
AlarmState: NEW
Acknowledged: FALSE
UserClearable: FALSE
Location:
AlarmAge: 0
NotificationData:
ProbableCause: Issue with Network Device
IfName: XXXXXXXXXXX
IfDesc: XXXXXXXXXXX
IfAlias: XXXXXXXXXXX
TicketStatus:
CRQ_ID:
Maintenance:
EnrichmentInfo:
============================================================

Search I'm using for the Clear Case (thats already a drilldown search from a bar chart in which a counting of alarms cleared in a bucket is presented)

index=general sourcetype=alarming (Event_Type=SET OR Event_Type=CLEARER) | transaction AlarmID startswith=Event_Type=SET endswith=Event_Type=CLEARED | eval duration_bucket=case(duration<=10, "0-10sec", duration=0, "0sec", duration>10 AND duration<=30, "10-30sec", duration>30 AND duration<=60, "30-60sec", duration>60 AND duration<=120, "60-120sec", duration>120 AND duration<=180, "120-180sec", duration>180 AND duration<=240, "180-240sec", duration>240 AND duration<=300, "240-300sec", duration>300, ">300sec") | search duration_bucket=180-240sec | table _time, DeviceName, AlarmTitle, duration

Re: Duration between two events with conditions

ITWhisperer — Tue, 06 Jul 2021 13:22:44 GMT

Try something like this

index=general sourcetype=alarming (Event_Type=SET OR Event_Type=CLEAR OR Event_Type=UPDATE) | fields _time AlarmID TTID Event_Type | eval inctime=if(match(TTID,"INC"),_time,null()) | eval cleartime=if(Event_Type="CLEAR",_time,null()) | eval TTID=if(TTID="",null(),TTID) | eval setevent=if(Event_Type="SET",1,null()) | sort 0 _time | streamstats sum(setevent) as setevent by AlarmID | stats first(inctime) as inctime first(_time) as settime first(TTID) as TTID first(cleartime) as cleartime by AlarmID setevent

This assumes that there is only one set and one clear per alarm at any one time.