Splunk Search

Search/Job missing in job_manager

dwong-rtr
Explorer

I currently do a search monthly for searches/jobs that take a long time. I then look up the job and if there is an alert associated with it. I will then work with teams/individuals to see if the job can be measured as a metric to trigger in SignalFX instead. However, there are some jobs I cannot find, and I think I am missing some part of the logic of how things work.

I run this search 
 

 

 

index=_internal sourcetype=scheduler user!=nobody | table _time user savedsearch_name status scheduled_time run_time result_count | sort run_time desc

 

 

Results look like this

dwongrtr_1-1722535487377.png

I try to look up the savedsearch_name  (like Weekly Sort Options) on this page -splunkcloud.com/en-GB/app/search/job_manager
but nothing comes up. 

dwongrtr_0-1722535319510.png

Some savedsearch_name show up with results, others don't and I'm not sure why?

0 Karma

dwong-rtr
Explorer

So just to close the loop- after some deep diving by a splunk support rep the issues that caused me to not see all the jobs on that page were due to:

1- permission levels- I was not an ADMIN and so several private jobs were not visible to me

2- incorrect expectations on how the page should work. I assumed it should show all jobs from x amount of time, regardless of how often a job runs or any other job attributes. The page was intended be used real time and not so much for historical runs like from a month ago.

3- TTL - each job has a life expectancy that determines how long its visible on that page. The calculations are convoluted and not obvious. So some jobs might show for longer vs others depending on various things like 

  • scheduled search should adhere to that dispatch.ttl
  • adhoc search should default to 10 minutes (no matter how long you are searching back
  • alert should have a TTL that depends on the action that it is taking

4- a bug that created a 2025 expiry date on the TTL for some of my searches/jobs which contributed to the confusion as to why some jobs show and others dont.- Rep was unable to determine cause of the 2025 bug.

marnall
Motivator

You seem to have the "App" filter set to Search & Reporting. Are you sure that the "Weekly Sort Options" savedsearch is in the Search & Reporting app? If not then you can set App to "All" to filter for all searches visible to you.

0 Karma

dwong-rtr
Explorer

OK tried that and still no results-

Screenshot 2024-08-05 at 10.12.32 AM.png

0 Karma

dwong-rtr
Explorer

I also sorted by Owner ascending and going to page 2 where the user shows up and the job actually doesn't show up in the job manager page. I also tried to filter by the user and they don't show up as an option. I can only filter by me as Owner or All.

0 Karma
Get Updates on the Splunk Community!

Automatic Discovery Part 1: What is Automatic Discovery in Splunk Observability Cloud ...

If you’ve ever deployed a new database cluster, spun up a caching layer, or added a load balancer, you know it ...

Real-Time Fraud Detection: How Splunk Dashboards Protect Financial Institutions

Financial fraud isn't slowing down. If anything, it's getting more sophisticated. Account takeovers, credit ...

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...