Using Machine Learning Toolkit to detect server la...

Splunk Search

I have key value that call (duration) in my application log that show duration of each job done.

each day when I get maximum duration it has lot’s of false positive because it is natural to become high duration in some point.

It’s not normal when it continues high duration.

e.g.
normal condition:

00:01:00.000 WARNING duration[0.01]
00:01:00.000 WARNING duration[100.01]
00:01:00.000 WARNING duration[0.01]

abnormal condition:

00:01:00.000 WARNING duration[0.01]
00:01:00.000 WARNING duration[100.01]
00:01:00.000 WARNING duration[50.01]

00:01:00.000 WARNING duration[90.01]
00:01:00.000 WARNING duration[100.01]
00:01:00.000 WARNING duration[0.01]

1-how can I detect abnormal condition with splunk? (Best way with minimum false positive on hug data)

2-which visualization or chart more suitable to show this abnormal condition daily? this is huge log file and it is difficult to show all data for each day on single chart.

Any idea?

Thanks,

Get Updates on the Splunk Community!

Using Machine Learning Toolkit to detect server latency

field extraction

regex

rex

stats

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Announcing Modern Navigation: A New Era of Splunk User Experience

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

Join the Conversation