Splunk Search

How can I create an alert for long running jobs?

thejasplunk67
Engager

Hi there,

Search to trigger an alert when the particular job (scheduled jobs) is running more than the threshold time (lets say 5 mins).
I have fields to create an alert are Job name (unique),  source, source type, index and time.

Thanks in Advance,


Regards,
Theja

Labels (2)
0 Karma
1 Solution

sjringo
Communicator

If what I provided is working for you please accept as a solution.

Then you can PM me with any specific questions and I will try my best to explain.

View solution in original post

0 Karma

thejasplunk67
Engager

Hi All,

Is there any possibility to send an alert to the user when second time search, I mean we want to skip (To confirm) the first time search alert




Thanks and Regards,
Theja 

0 Karma

jdunlea
Contributor

You could run a search on a schedule that checks the status of current jobs using the search/jobs API endpoint and then look at the runDuration for those jobs and see when they are running more than a certain threshold. 

 

Something like the following will get you started. 

| rest /services/search/jobs search="delegate=scheduler" | fields label savedSearchLabel published dispatchState runDuration delegate

 

Remember, there are lots more fields available for the results returned from the search/jobs API endpoint, so you can always rework the search to use some of those other fields also. 

thejasplunk67
Engager

Hi jdunlea,

Thanks for the quick response, 

How can I calculate the time while running the running the job?, I have time filed to see when the job is running but struggling to calculate the time of running job and parallelly checking with the threshold time weather running job is crossing.

Note:  Unique field i have job name alone, and to calculate time we have time field ( Formate eg: 9/18/22
1:20:04.585 AM)



Thanks and Regards, 
Theja

0 Karma

jdunlea
Contributor

What do you mean the "threshold" time? How are you calculating this?

If a search is configured to run once per hour over 60 mins of data, what would the threshold time be in this case? 60 mins?

And if the search is configured to run once per hour looking over 2 hours of data, is the threshold time still 60 mins? 

0 Karma

sjringo
Communicator

The original post said 5 Minutes which is 300 seconds.

Threshold is not a calculated value and does not matter on lookback time ?

Im guessing the assumptions I made are different than yours. Event data would help clarify any of those...

 

0 Karma

sjringo
Communicator

Hi,

I have been working on something very similar.  I am somewhat new so this might not be the 'best' solution but it will work.  Replace 300 with your threshold value.  Then when setting up your alert you can trigger an action if results are > 0.

index=yourIndex sourcetype=yourSourceType ("START of JobString") earliest=-1d@d

| eval startTime=_time

| eval timeNow=now()

| eval duration = timeNow - startTime

| eval timeStringDuration = strftime(duration, "%H:%M:%S")

| eval timeStringStart = strftime(startTime, "%H:%M:%S")

| eval timeStringNow = strftime(timeNow, "%H:%M:%S")

| where duration > 300

| table startTime, timeStringStart, timeNow, timeStringNow, duration, timeStringDuration

0 Karma

thejasplunk67
Engager

Hi @sjringo,

Thanks a lot, seems like it is working, Could you please help me to understand how it works

Thanks in Advance,

Regards,
Theja Boddhaluru 

0 Karma

sjringo
Communicator

If what I provided is working for you please accept as a solution.

Then you can PM me with any specific questions and I will try my best to explain.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...