- Find Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- Why are jobs showing as Dispatched at 12/31/69 and...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?
1969 ... wow, what a year!
We have noticed a problem where we find users complaining that they can't search, and in most cases, their work is all queued. They often have one or more running jobs where the Dispatched at date/time is 12/31/69 7:00:00 PM.
This usually means they are trying to search all-time.
Is there are a way to search and alert on this condition?
We would like to find these as they are happening rather than after the user is bogged down.
Also, any tips on debugging this?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
After upgrading to 6.4.1, these seem to have disappeared for us.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Snoobzilla, did you ever find a reasonable solution to this problem? It seems that it has begun happening to me just within the past day or so.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I suspect these are scheduled jobs that have not started yet (and should have) so if your instance is running behind on scheduled jobs these start to stack up... which of course is exactly when you are most likely to be looking at activity, when things are slow. Suspicion is based on feedback from related case we are working through with Splunk.
Take a look at your skipped jobs ratio at the same time you are seeing these in DMC app/splunk_management_console/scheduler_activity_deployment
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There is a rt search that Splunk decides to launch when a user loads the Home page. So your users do not intentionally run this search, Splunk slid in in by default for silly reasons, IMO.
Luckily, in recent versions this can be disabled:
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The very first thing that I do and suggest that all of my clients do is to COMPLETELY DISABLE realtime by removing it from all roles and deleting it from the TimePicker settings. It is too dangerous and tricky to be let loose except in very VERY specific (and highly uncommon) use cases. Kill realtime dead and all of these problems might go away.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is simply not the case. Real-time searches do not coincide with these issues (which we are seeing as well). These problems simply cannot be attributed to real-time searches.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Will have admin check, but I think we killed real time.
We do have accelerated data models and reports.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I get jobs dispatched at 12/31/69 7:00:00 PM, but I was not attempting to search all-time, I was using a relative 7 Day search starting at midnight (to be specific earliest=-7d@d latest=now). Once one of these appears, every attempt afterwards has this same 1969 timestamp.
I usually have to Finalize each one, then Delete it afterwards, wait a few minutes and try again.
Oh, by the way, it would not matter if it was -7days at midnight or -7days from this time, the issue appears somewhat randomly and stays for a few minutes to an hour before it goes away. It also occurs when I'm just doing a |inputlookup Tablename.csv and using the default relative 24hour time, that same 1969 timestamp in the Dispatched At field in the Jobs area just appears, and everything is on hold until I can Finalize, Delete, and retry in a few minutes (sometimes repeatedly).
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Someone else reported... https://answers.splunk.com/answers/426695/why-i-have-very-old-time-in-my-job-list.html#answer-432343
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Any resolution? We have 2200 of these "jobs".
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
| [earliest time, latest time]
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
| [earliest time, latest time]
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
| [earliest time, latest time]
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
| [earliest time, latest time]
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
| [earliest time, latest time]
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
| [earliest time, latest time]
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
| [earliest time, latest time]
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
| [earliest time, latest time]
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
| [earliest time, latest time]
12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Walt, I got all excited when this came up in google. 😛
How to Monitor Google Kubernetes Engine (GKE)
Index This | How can you make 45 using only 4?
Splunk Education Goes to Washington | Splunk GovSummit 2024
