How to compare two log messages and send an alert email if the second message does not come within 5 minutes of the first?

[aniketb]

I'm looking to set up an alert if my server started correctly.

First log message: java org.apache.catalina.startup.Catalina
Second log message: org.apache.catalina.startup.Catalina start

For a specific sourcetype/host/source, if the second message does not come in 5 minutes of the first, I want Splunk to email me.

How do I go about doing that?

[renjith_nair]

Try this,

Hope you have an extracted field for messages. If not, extract using rex which will make things easy

index="your index" ("java org.apache.catalina.startup.Catalina" OR "org.apache.catalina.startup.Catalina start") 
|fields _time,Log_message_field|stats earliest(_time) as start,latest(_time) as end,earliest(Log_message_field) as FirstMessage,latest(Log_message_field) as LastMessage|eval TimeDiff=end-start|search (TimeDiff > 300 OR NOT LastMessage="org.apache.catalina.startup.Catalina start")

Alternatively you can try transaction also

index="your index " host="your host" ("java org.apache.catalina.startup.Catalina" OR "org.apache.catalina.startup.Catalina start")  |transaction startswith="catalina.startup.Catalina" Endswith="org.apache.catalina.startup.Catalina start"|where duration > 300

[aniketb]

Hi Renjith,

Thank you for your response.

For suggestion 2, I tried in scenario where I didn't get second message (server never started) and even tried lowering the duration from 300 to 50 but didn't get any result. (I did fill index and host correctly)

For suggestion 1, shouldn't both log lines be extracted separately for comparison. What are we expecting in Log_message_field? Here are my complete logs if it helps:

Entry 2:
Feb 14, 2016 10:31:16 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 304361 ms

Entry 1:
2016-02-14 22:26:09 [00000002] info [java ] Added missing MBeanServer with default domain: DefaultDomain
usage: java org.apache.catalina.startup.Catalina [ -config {pathname} ] [ -nonaming ] { -help | start | stop }

[renjith_nair]

Sorry, I have replied a day ago but seems like i missed the submit button 🙂

Are these entries from splunk or your log file because the timestamp format is different?

For suggestion 2, try this and see if the transactions are able to match

 index="your index " host="your host" ("java org.apache.catalina.startup.Catalina" OR "org.apache.catalina.startup.Catalina start")  |transaction startswith="catalina.startup.Catalina" Endswith="org.apache.catalina.startup.Catalina start"

For suggestion 1, Log_message_field is java org.apache.catalina.startup.Catalina and org.apache.catalina.startup.Catalina start