How to create an alert if log message does not show up within 10 minutes of the "service stopped" log message?

[santosh1]

Hi There,

I have two Application log messages that I receive in Splunk 

1. Service stopped

2. Service Started

I need to create an alert if the "service started" log message does not show up within 10 minutes of the "Service  stopped" log message.

So the alert needs to trigger an email only if it has been more than 10 min since the service stopped an a new log message stating Service started does not show up in the logs.

I am finding some solutions here, but need one that will compare the log messages, I am new to splunk please do share the syntax as I would not know how to work it out without it.

index=* | search app=xxx log="xxx" message="*service stopped/started*"

[santosh1]

yes, I will be putting the index name, I added * just to give an example.

I will have the log message saying service started and service stopped, it will have some values before and after , but the messages will definitely give the status.

to give you an exact Idea it will be 

1. data service started and process 1.

2.Data service stopped.

these are the two messages exactly.

[gcusello]

Hi @santosh1,

if in the message field you have the string "service stopped" or the string "service started", you could run something like this:

index=* app=xxx log="xxx" (message="service stopped" OR message="service started")
| message_kind=if(like(message,"%service stopped%"),"service stopped","service stopped")
| stats dc(message_kind) AS dc_message_kind values(message_kind) As message_kind earliest(_time) AS earliest latest(_time) AS latest
| search (dc_message_kind=2 AND latest-earliest>600) OR (dc_message_kind=1 AND message_kind="service stopped")

One additional hint: it's an erro to have index=* | search ..., because search best practices say that it's better to put all search parameters as left as possible.

Ciao.

Giuseppe

[santosh1]

I am receiving an unknown search command 'message' error when I run the search 

[gcusello]

Hi @santosh1,

sorry, I forgot an eval command and I used the strings you shared, please try this:

index=* app=xxx log="xxx" ("data service started" OR "Data service stopped")
| eval message_kind=if(searchmatch("Data service stopped"),"service stopped","service started")
| stats dc(message_kind) AS dc_message_kind values(message_kind) As message_kind earliest(_time) AS earliest latest(_time) AS latest
| search (dc_message_kind=2 AND latest-earliest>600) OR (dc_message_kind=1 AND message_kind="service stopped")

About the additional hint I gave, it's related to the approach avoiding two steps search:

• before index=* (or whatever other index
• then | search app=xxx log="xxx" ...

It's better to put all thesearch condition in the main search, because using your approach the search before takes all results from the main search and then filter them using the second search, instead putting all the search conditions in the main search you have a faster search.

Ciao.

Giuseppe

[santosh1]

Hi @gcusello 

I had a question as to the condition of the alert, so basically when I create the alert I am setting the cron job as */5 * * * *

Time Range is default set to last 5m - Do I change this?

Alert type- scheduled

So , I need the alert to check the status and trigger if the service start message and service stop message have a 5minute delay

[gcusello]

Hi @santosh1,

it's always better to put additional questions in a new one so more people can help you better and quicker.

Anyway, the cron schedule is correct.

the time period could be correct, you should only check what's the delay of your data, maybe it could be better to use latest=-10m@m earliest=-5m@m to be sure that eventual delayed events will be considered.

About the condition of 5 minutes delay between stop and start is considere in my answer.

Ciao.

Giuseppe