topic Re: monitor failed jobs in Splunk Enterprise

monitor failed jobs

sarit_s — Sun, 11 Oct 2020 10:45:23 GMT

Hello

i want to create a dashboard that monitors the failed job (savedsearches the i can see in the activity page)

how can search for failed jobs ?
i saw that ITSI can do it but i wonder if there is a way to do it with Splunk itself

thanks

Re: monitor failed jobs

inventsekar — Sun, 11 Oct 2020 12:15:53 GMT

Hi @sarit_s by Splunk also, its possible.

the applications logs are already ingested to splunk, we hope. and if you are not sure how to search for the failed jobs, then, maybe, you start from the hostname. simply search for the host which is running the app.

then, from the events list, you can find out the source/sourcetype. you may need to do field extractions using the rex command.

let us know how your search goes, so that we can help you. thanks.

Re: monitor failed jobs

sarit_s — Sun, 11 Oct 2020 12:19:23 GMT

In the jobs list i see the inpect job option. I see there field name dispatchedStatus but i cant search for it

is there a way to search something like this ?

Re: monitor failed jobs

inventsekar — Sun, 11 Oct 2020 12:33:45 GMT

i assumed that "failed jobs" meaning, some app related failed jobs.

for splunk's failed jobs,

Splunk GUI---->Activity---> Jobs---> Status dropdown, select Failed.

or, the page URL is(pls update your splunk link on this below URL)

https://yourcompany.splunk.com/en-US/app/search/job_manager?owner=&jobStatus=failed

Re: monitor failed jobs

inventsekar — Sun, 11 Oct 2020 12:43:41 GMT

Hi Sarit, Please check these search queries:

| rest /services/search/jobs
| table dispatchState *.search

Re: monitor failed jobs

sarit_s — Sun, 11 Oct 2020 13:05:42 GMT

I know it is there

i want to monitor this jobs and add it as dashboard panel

is there a query that can give me the results ?

Re: monitor failed jobs

sarit_s — Sun, 11 Oct 2020 13:29:20 GMT

the first one looks close to what i want. is there a way to filter so i will see only failed jobs ?

Re: monitor failed jobs

inventsekar — Sun, 11 Oct 2020 14:48:22 GMT

| rest /services/search/jobs isFailed=1

this will list only failed jobs.

if this solved your query, please accept this as the solution.

Re: monitor failed jobs

sarit_s — Sun, 11 Oct 2020 19:54:30 GMT

well, there is something weird

even thought im searching for isFailed=1 im getting results of isFailed=0

image below

Re: monitor failed jobs

inventsekar — Sun, 11 Oct 2020 20:28:09 GMT

Check...

| rest /services/search/jobs | where isFailed=1

| rest /services/search/jobs | search isFailed=1

PS - Karma points are appreciated, if a reply solved your problem, please accept it as the solution.