This is really strange.I am seeing in the job manager that there are many jobs running with created at date 12/31/69 with runtime waiting and status running .I tried deleting those jobs but they kept on coming back .How to resolve this issue .We dont have any real time searches running.I changed all of them to scheduled.
Thanks in Advance
Check if there is any "All Time" scheduled real-time search running/configured on that instance.
I checked for real time searches and there are none.How can I check specifically for searches running all time?
Give this a try
| rest timeout=0 splunk_server=local /servicesNS/-/-/saved/searches search="is_scheduled=1 disptach.earliest_time=0" | table title eai:acl.owner eai:acl.app dispatch.*t_time search cron_schedule
Results with dispatch.latest_time as blank OR "now" are the searches running with all time timerange
There is nothing ...no results.The jobs that are having those Dispatched at 12/31/69 .These are all searches made from API calls.When these API calls were stopped I dont see these jobs.BUt when the API calls start I see the jobs again
I checked the jobs that are running with that date .I see the search like below with indexearliest and indexlatest ..what does this mean ?
index=abc sourcetype=xyz earliest=-4h host=abbb indexearliest=1581099754 indexlatest=1581099799 | sort 0 +indextime | eval message=raw | table raw,indextime,host
Those are the date/time ranges of the search, in Unix timestamp.
1581099754 = Friday, February 7, 2020 12:22:34 PM (CST)
1581099799 = Friday, February 7, 2020 12:23:19 PM (CST)
Thank you .What would be the solution for me to implement to make the jobs going to run on that wierd date and they dont have runtime waiting and status running
Splunk has two default timestamp field added to each event, time which is the time when event occurred (set based on timestamp parsing rules) and _indextime which is the time when Indexers stored the data into Splunk. The timerange picker (and earliest/lasted filters in search) filters the data based on _time, e.g. Events happening in last 60 minutes. The _indexearliest and indexlatest timerange modifier filter data based on indextime value. Generally the timerange for searches using `index_*` modifier is quite high, so that all the possible events are within the range. The timerange in those searches can be "All time".
So are these searches causing the issue that I am looking for and what would I need to do in order to get these get to normal state