Splunk Search

Detect abnormalities with splunk

indeed_2000
Motivator

Hi

I have key value that call (duration) in my application log that show duration of each job done.

each day when I get maximum duration it has false positive because it is natural to become high duration in some point.

It’s not normal when it continues high duration.

e.g. 
normal condition:

00:01:00.000 WARNING duration[0.01]
00:01:00.000 WARNING duration[100.01]
00:01:00.000 WARNING duration[0.01]

 

abnormal condition:

00:01:00.000 WARNING duration[0.01]
00:01:00.000 WARNING duration[100.01]
00:01:00.000 WARNING duration[50.01]

00:01:00.000 WARNING duration[90.01]
00:01:00.000 WARNING duration[100.01]
00:01:00.000 WARNING duration[0.01]

 

1-how can I detect abnormal condition with splunk? (Best way with minimum false positive on hug data)

2-which visualization or chart more suitable to show this abnormal condition daily? this is huge log file and it is difficult to show all data for each day on single chart.

Any idea?

 Thanks,

 

Labels (5)
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...