Splunk Search

Detect abnormalities with splunk

indeed_2000
Builder

Hi

I have key value that call (duration) in my application log that show duration of each job done.

each day when I get maximum duration it has false positive because it is natural to become high duration in some point.

It’s not normal when it continues high duration.

e.g. 
normal condition:

00:01:00.000 WARNING duration[0.01]
00:01:00.000 WARNING duration[100.01]
00:01:00.000 WARNING duration[0.01]

 

abnormal condition:

00:01:00.000 WARNING duration[0.01]
00:01:00.000 WARNING duration[100.01]
00:01:00.000 WARNING duration[50.01]

00:01:00.000 WARNING duration[90.01]
00:01:00.000 WARNING duration[100.01]
00:01:00.000 WARNING duration[0.01]

 

1-how can I detect abnormal condition with splunk? (Best way with minimum false positive on hug data)

2-which visualization or chart more suitable to show this abnormal condition daily? this is huge log file and it is difficult to show all data for each day on single chart.

Any idea?

 Thanks,

 

Labels (5)
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!