Monitoring Splunk

I'm getting a "The running job 'id' was canceled remotely or expired" error

werye
Engager

The machine the Splunk server was running on crashed. Once we rebooted it and started Splunk, all searches return empty results and display the running job was canceled remotely or expired error. I get the same results even if I use remote desktop and run the browser on the Splunk server. Any idea how to fix this?

Tags (1)
1 Solution

the_wolverine
Champion

Posting here because this may be useful to someone else. Support was able to reproduce the condition when the system clock was off between indexers/search head (240 seconds offset).

Check your system clocks and make sure they are all in sync. (Use NTP)

View solution in original post

the_wolverine
Champion

Posting here because this may be useful to someone else. Support was able to reproduce the condition when the system clock was off between indexers/search head (240 seconds offset).

Check your system clocks and make sure they are all in sync. (Use NTP)

jmaslowski
Engager

it really helped, thank you!

0 Karma

the_wolverine
Champion

Is this resolved? What version were you running? This happened to us immediately after upgrading the search head to version 4.2.5.6.

0 Karma

shadinaif
New Member

I have had the same error this month while exporting to csv using outputcsv command. The problem was related actually to free disk space but not logged in Splunk log files! I think that Splunk is using multiphase mechanism to accomplish indexing jobs; that’s why it needs more than double free space of the expected results size.
I have monitored the directory %SPLUNK%\var\run\splunk . There were many .LOCK files eating disk space while indexing job was running.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...