Splunk Search
Highlighted

Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Path Finder

1969 ... wow, what a year!

We have noticed a problem where we find users complaining that they can't search, and in most cases, their work is all queued. They often have one or more running jobs where the Dispatched at date/time is 12/31/69 7:00:00 PM.

This usually means they are trying to search all-time.

Is there are a way to search and alert on this condition?
We would like to find these as they are happening rather than after the user is bogged down.

Also, any tips on debugging this?

Highlighted

Re: Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Builder

Walt, I got all excited when this came up in google. 😛

0 Karma
Highlighted

Re: Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Builder

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

| [earliest time, latest time]

12/31/69 7:00:00 PM ithomas EventKNow 0.13MB 0 00:00:00 Oct 29, 2015 10:38:04 AM Running (0%) Inspect | Save | Pause | Finalize | Delete

0 Karma
Highlighted

Re: Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Path Finder

Any resolution? We have 2200 of these "jobs".

0 Karma
Highlighted

Re: Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Path Finder

I get jobs dispatched at 12/31/69 7:00:00 PM, but I was not attempting to search all-time, I was using a relative 7 Day search starting at midnight (to be specific earliest=-7d@d latest=now). Once one of these appears, every attempt afterwards has this same 1969 timestamp.

I usually have to Finalize each one, then Delete it afterwards, wait a few minutes and try again.

Oh, by the way, it would not matter if it was -7days at midnight or -7days from this time, the issue appears somewhat randomly and stays for a few minutes to an hour before it goes away. It also occurs when I'm just doing a |inputlookup Tablename.csv and using the default relative 24hour time, that same 1969 timestamp in the Dispatched At field in the Jobs area just appears, and everything is on hold until I can Finalize, Delete, and retry in a few minutes (sometimes repeatedly).

Highlighted

Re: Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Builder
0 Karma
Highlighted

Re: Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Esteemed Legend

The very first thing that I do and suggest that all of my clients do is to COMPLETELY DISABLE realtime by removing it from all roles and deleting it from the TimePicker settings. It is too dangerous and tricky to be let loose except in very VERY specific (and highly uncommon) use cases. Kill realtime dead and all of these problems might go away.

0 Karma
Highlighted

Re: Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Builder

Will have admin check, but I think we killed real time.

We do have accelerated data models and reports.

0 Karma
Highlighted

Re: Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Path Finder

This is simply not the case. Real-time searches do not coincide with these issues (which we are seeing as well). These problems simply cannot be attributed to real-time searches.

0 Karma
Highlighted

Re: Why are jobs showing as Dispatched at 12/31/69 and keep running? How to search for and alert on this condition?

Champion

There is a rt search that Splunk decides to launch when a user loads the Home page. So your users do not intentionally run this search, Splunk slid in in by default for silly reasons, IMO.

Luckily, in recent versions this can be disabled:

https://answers.splunk.com/answers/141179/how-to-remove-automatic-real-time-searches-that-run-when-u...

0 Karma