topic Re: Create Splunk alerts for cron jobs in Alerting

Create Splunk alerts for cron jobs

gpunjabi — Thu, 14 Feb 2019 10:06:45 GMT

I want to create a Splunk alert for Cron job it will trigger an alert when cron job is not successful or not ran? Any one can help me on this.

Re: Create Splunk alerts for cron jobs

burwell — Tue, 29 Sep 2020 23:14:31 GMT

Hi @gpunjabi

Are you asking about scheduled jobs in Splunk and finding out the status?

If so..

index=_internal source="/opt/splunk/var/log/splunk/scheduler.log" status!=delegated* status!=success
| stats count by savedsearch_name,status

Will show you failed scheduled jobs.

If you have a search head cluster with a deployer, you can get a lot more information about scheduled jobs including the reason for your jobs, I use the Monitoring Console. I wrote about using it here: https://answers.splunk.com/answers/514181/skipped-searches-on-shc.html

You can find skipped jobs, long running jobs etc.

Re: Create Splunk alerts for cron jobs

woodcock — Fri, 15 Feb 2019 02:04:38 GMT

This is the "skipped search" problem. Configure a Monitoring Console and run the Health Checks. Look at the results of the Skipped Searches one. You can steal this search and use it as an alert.

Re: Create Splunk alerts for cron jobs

richgalloway — Fri, 15 Feb 2019 13:44:36 GMT

First you need to index your cron logs into Splunk. Then you can run a search for the name of the job that has to run. Schedule the search to run a couple of minutes after the cron job is expected to complete. If the search returns no results, trigger an alert.

Re: Create Splunk alerts for cron jobs

gpunjabi — Sat, 16 Feb 2019 15:50:41 GMT

Hello Richgalloway,
I have one scenario mentioning below if you can help me on this.
have required like this Create the Splunk alert for following jobs after deployment to UAT and Production environment to ensure we have the alerts present in system for each jobs.
Make them active in UAT to test if they trigger on correct condition.
On production they can be turned ON when market starts using these jobs.

Cron Job is full-wcpIndexAT-cronjob- that runs 3:05 AM Every Day

Re: Create Splunk alerts for cron jobs

richgalloway — Sat, 16 Feb 2019 22:06:05 GMT

Are you indexing the cron log (/var/log/cron)? Does it contain an event when full-wcpIndexAT-cronjob- completes? If the answer to both of those questions is yes, then you can schedule a search to run at 3:10 am every day that searches for the job completion string. If the string is not found, send an alert.

Re: Create Splunk alerts for cron jobs

gpunjabi — Mon, 18 Feb 2019 10:10:43 GMT

Hello Rich,

Thank you for your answer, We wan to trigger an alarm when Job gets failed, We have already configure this in our aws cloudwatch monitoring need to integrate with splunk now.

Re: Create Splunk alerts for cron jobs

gpunjabi — Tue, 29 Sep 2020 23:23:25 GMT

Hi Rich, I have written a query for a single cron job but I want to use regex and make this query so that I can run for others jobs also.
sourcetype=hybris_console "full-wcpIndexCA-cronjob" earliest=-80m | rex field=_raw "[(?\w+-\w+-\w+)::" | eval Period=if(_time>=relative_time(now(),"-24h"),1,2) | stats min(Period) as periodMin max(Period) as periodMax by cronType | eval stopped = if(periodMin=2,"true", "false") | eval restarted = if(stopped="false" AND periodMax=1,"true", "false") | where stopped="true" OR restarted="true" | table cronType stopped restarted.

Re: Create Splunk alerts for cron jobs

richgalloway — Fri, 22 Feb 2019 13:30:06 GMT

If the event text is similar for each cron job, this should work. I wonder only about the "full-wcpIndexCA-cronjob" part - is that specific to one job?

Re: Create Splunk alerts for cron jobs

gpunjabi — Sat, 23 Feb 2019 16:22:06 GMT

Yes Rich it specific to CA cron job we have multiple cron jobs
1.i.e full-wcpIndexDE-cronjob
full-wcpIndexBE-cronjob
full-wcpIndexIT-cronjob
full-wcpIndexSE-cronjob
full-wcpIndexPT-cronjob
full-wcpIndexNO-cronjob

2.Also I want to check if the alarm run in the last 24 h in this query.

I have tried replacing full-wcpIndex[A-Z]{2}-cronjob in the query this seems to be not working. Can you please help me with the above points.

Re: Create Splunk alerts for cron jobs

gpunjabi — Sat, 23 Feb 2019 16:38:31 GMT

Hi Rich, any how my comment is not showing when I am trying to post my comment, apologies if your are receiving the notification.

Re: Create Splunk alerts for cron jobs

richgalloway — Sat, 23 Feb 2019 17:05:46 GMT

Try sourcetype=hybris_console "full-wcpindex*-cronjob" earliest=-80m | rex field=_raw "[(?\w+-\w+-\w+)\::" | eval Period=if(_time>=relative_time(now(),"-24h"),1,2) | stats min(Period) as periodMin max(Period) as periodMax by cronType | eval stopped = if(periodMin=2,"true", "false") | eval restarted = if(stopped="false" AND periodMax=1,"true", "false") | where stopped="true" OR restarted="true" | table cronType stopped restarted. You will need to fix the rex command as it was garbled in your comment.
One cannot use regex meta-characters in a base search, only in certain command like rex, regex, replace, and match.

Re: Create Splunk alerts for cron jobs

richgalloway — Sat, 23 Feb 2019 17:06:16 GMT

Your comment was sent for moderation since your Karma score is still low.

Re: Create Splunk alerts for cron jobs

gpunjabi — Sat, 23 Feb 2019 18:05:21 GMT

It is throwing an Error in 'rex' command: Encountered the following error while compiling the regex '[(?\w+-\w+-\w+)::': Regex: invalid range in character class in I think "\[(?<cronType>\w+\-\w+\-\w+)\::" this will work. Sorry I forgot to add the regex part in the mail it was actually there, Is it fine now Rich?

Re: Create Splunk alerts for cron jobs

richgalloway — Sat, 23 Feb 2019 19:23:18 GMT

Does it work?

Re: Create Splunk alerts for cron jobs

gpunjabi — Sun, 24 Feb 2019 19:04:12 GMT

Yeah Rich, the query works and it is showing the cronjobs that are present in the log, but I want to make an alarm from this now. but I am not sure whether it is an failure result or not.

Re: Create Splunk alerts for cron jobs

richgalloway — Tue, 26 Feb 2019 04:22:10 GMT

Once you have your search for cron jobs, add a where clause for the number of expected results. ... | stats count | where count < x. Trigger the alarm if the number of results is not zero.
You could also compare the list of jobs to an expected list in a lookup file. If any jobs are missing you'll have results. Trigger the alarm if there are any.

Re: Create Splunk alerts for cron jobs

gpunjabi — Tue, 29 Sep 2020 23:32:33 GMT

Hi Rich, My earlier query seems to be complex .I need to simplfy query as much as I can , so I need to trigger my search as an event. I can trigger this event at 7:00 AM So that it can show failure (Importantly ) of all cronjobs . Usually all Jobs start from 2:45 AM and goes till 5:55 AM after all jobs are completed events get triggered and I can see the stats.

sourcetype=hybris_console "full-wcpIndexCA-cronjob" earliest=-80m | rex field=_raw "\[(?<cronType>\w+\-\w+\-\w+)\::" | eval Period=if(_time>=relative_time(now(),"-24h"),1,2) | stats min(Period) as periodMin max(Period) as periodMax by cronType | eval stopped = if(periodMin=2,"true", "false") | eval restarted = if(stopped="false" AND periodMax=1,"true", "false") | where stopped="true" OR restarted="true" | table cronType stopped restarted

if you can help me in simplyfying my query with the above condition

Also I didn't get your previous comment solution.

Sample Failure event

3/2/19
4:25:07.538 AM

INFO | jvm 1 | main | 2019/03/02 04:25:07.538 | [m[33mWARN full-wcpIndexFR-cronjob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob [SolrIndexerJob] Error during indexer call: frIndex
host = uat_hybris_system_0 source = /opt/sap/hybris/log/tomcat/console-20190302.log

Image url- [1]: https://storage.googleapis.com/splunk-alert/Screen%20Shot%202019-03-02%20at%209.48.27%20PM.png

I would request you to kindly help me here and will wait for your response.

Re: Create Splunk alerts for cron jobs

richgalloway — Sun, 03 Mar 2019 20:47:53 GMT

The best way to improve your query is to add an index specification before the first |.

Using the resulting query, you can save it as an alert and have the alert trigger when the number of results is fewer than the expected number of cron jobs.
(You can disregard my last comment.)

Re: Create Splunk alerts for cron jobs

gpunjabi — Tue, 29 Sep 2020 23:32:48 GMT

Okay, Rich , I have one more Idea, I am not sure whether It will work or not, if we can can catch aborted job Error Message from _raw feel and use error message as an alert. Will it be a right approach, I am not sure, I have written one sample query with error message filed in the search field and use as an alert for all cron jobs.

Also the above query is only for CA job, I want that it can work for all cron jobs, the above query I checked with full-wcpIndex*-cronjob in the old query it not showing a proper output.

The query is
index=leap_uat_logs sourcetype=hybris_console "full-wcpIndex*-cronjob" "Error during indexer call: Index" NOT debug source=/opt/sap/hybris/log/tomcat/console

The sample result.

3/2/19
4:25:07.538 AM
INFO | jvm 1 | main | 2019/03/02 04:25:07.538 | [m[33mWARN full-wcpIndexFR-cronjob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob [SolrIndexerJob] Error during indexer call: frIndex
host = uat_hybris_system_0 source = /opt/sap/hybris/log/tomcat/console-20190302.log

Thanks
Gaurav