topic Re: How can I create an alert for long running jobs? in Splunk Search

How can I create an alert for long running jobs?

thejasplunk67 — Thu, 06 Oct 2022 13:53:15 GMT

Hi there,

Search to trigger an alert when the particular job (scheduled jobs) is running more than the threshold time (lets say 5 mins).
I have fields to create an alert are Job name (unique), source, source type, index and time.

Thanks in Advance,

Regards,
Theja

Re: How can I create an alert for long running jobs?

jdunlea — Thu, 06 Oct 2022 14:43:19 GMT

You could run a search on a schedule that checks the status of current jobs using the search/jobs API endpoint and then look at the runDuration for those jobs and see when they are running more than a certain threshold.

Something like the following will get you started.

| rest /services/search/jobs search="delegate=scheduler" | fields label savedSearchLabel published dispatchState runDuration delegate

Remember, there are lots more fields available for the results returned from the search/jobs API endpoint, so you can always rework the search to use some of those other fields also.

Re: How can I create an alert for long running jobs?

thejasplunk67 — Fri, 07 Oct 2022 10:00:03 GMT

Hi jdunlea,

Thanks for the quick response,

How can I calculate the time while running the running the job?, I have time filed to see when the job is running but struggling to calculate the time of running job and parallelly checking with the threshold time weather running job is crossing.

Note: Unique field i have job name alone, and to calculate time we have time field ( Formate eg: 9/18/22
1:20:04.585 AM)

Thanks and Regards,
Theja

Re: How can I create an alert for long running jobs?

sjringo — Fri, 07 Oct 2022 12:51:06 GMT

Hi,

I have been working on something very similar. I am somewhat new so this might not be the 'best' solution but it will work. Replace 300 with your threshold value. Then when setting up your alert you can trigger an action if results are > 0.

index=yourIndex sourcetype=yourSourceType ("START of JobString") earliest=-1d@d

| eval startTime=_time

| eval timeNow=now()

| eval duration = timeNow - startTime

| eval timeStringDuration = strftime(duration, "%H:%M:%S")

| eval timeStringStart = strftime(startTime, "%H:%M:%S")

| eval timeStringNow = strftime(timeNow, "%H:%M:%S")

| where duration > 300

| table startTime, timeStringStart, timeNow, timeStringNow, duration, timeStringDuration

Re: How can I create an alert for long running jobs?

jdunlea — Fri, 07 Oct 2022 14:42:06 GMT

What do you mean the "threshold" time? How are you calculating this?

If a search is configured to run once per hour over 60 mins of data, what would the threshold time be in this case? 60 mins?

And if the search is configured to run once per hour looking over 2 hours of data, is the threshold time still 60 mins?

Re: How can I create an alert for long running jobs?

sjringo — Fri, 07 Oct 2022 14:58:30 GMT

The original post said 5 Minutes which is 300 seconds.

Threshold is not a calculated value and does not matter on lookback time ?

Im guessing the assumptions I made are different than yours. Event data would help clarify any of those...

Re: How can I create an alert for long running jobs?

thejasplunk67 — Sat, 08 Oct 2022 14:11:13 GMT

Hi @sjringo,

Thanks a lot, seems like it is working, Could you please help me to understand how it works

Thanks in Advance,

Regards,
Theja Boddhaluru

Re: How can I create an alert for long running jobs?

sjringo — Sat, 08 Oct 2022 14:36:33 GMT

If what I provided is working for you please accept as a solution.

Then you can PM me with any specific questions and I will try my best to explain.

Re: How can I create an alert for long running jobs?

thejasplunk67 — Sun, 09 Oct 2022 11:31:41 GMT

Hi All,

Is there any possibility to send an alert to the user when second time search, I mean we want to skip (To confirm) the first time search alert

Thanks and Regards,
Theja