Alerting

How to compare two log messages and send an alert email if the second message does not come within 5 minutes of the first?

aniketb
Path Finder

I'm looking to set up an alert if my server started correctly.

First log message: java org.apache.catalina.startup.Catalina
Second log message: org.apache.catalina.startup.Catalina start

For a specific sourcetype/host/source, if the second message does not come in 5 minutes of the first, I want Splunk to email me.

How do I go about doing that?

0 Karma

renjith_nair
Legend

Try this,

Hope you have an extracted field for messages. If not, extract using rex which will make things easy

index="your index" ("java org.apache.catalina.startup.Catalina" OR "org.apache.catalina.startup.Catalina start") 
|fields _time,Log_message_field|stats earliest(_time) as start,latest(_time) as end,earliest(Log_message_field) as FirstMessage,latest(Log_message_field) as LastMessage|eval TimeDiff=end-start|search (TimeDiff > 300 OR NOT LastMessage="org.apache.catalina.startup.Catalina start")

Alternatively you can try transaction also

index="your index " host="your host" ("java org.apache.catalina.startup.Catalina" OR "org.apache.catalina.startup.Catalina start")  |transaction startswith="catalina.startup.Catalina" Endswith="org.apache.catalina.startup.Catalina start"|where duration > 300
---
What goes around comes around. If it helps, hit it with Karma 🙂

aniketb
Path Finder

Hi Renjith,

Thank you for your response.

For suggestion 2, I tried in scenario where I didn't get second message (server never started) and even tried lowering the duration from 300 to 50 but didn't get any result. (I did fill index and host correctly)

For suggestion 1, shouldn't both log lines be extracted separately for comparison. What are we expecting in Log_message_field? Here are my complete logs if it helps:

Entry 2:
Feb 14, 2016 10:31:16 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 304361 ms

Entry 1:
2016-02-14 22:26:09 [00000002] info [java ] Added missing MBeanServer with default domain: DefaultDomain
usage: java org.apache.catalina.startup.Catalina [ -config {pathname} ] [ -nonaming ] { -help | start | stop }

0 Karma

renjith_nair
Legend

Sorry, I have replied a day ago but seems like i missed the submit button 🙂

Are these entries from splunk or your log file because the timestamp format is different?

For suggestion 2, try this and see if the transactions are able to match

 index="your index " host="your host" ("java org.apache.catalina.startup.Catalina" OR "org.apache.catalina.startup.Catalina start")  |transaction startswith="catalina.startup.Catalina" Endswith="org.apache.catalina.startup.Catalina start"

For suggestion 1, Log_message_field is java org.apache.catalina.startup.Catalina and org.apache.catalina.startup.Catalina start

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...