Splunk Search

[EventsViewer module] Cannot access search data;; job 1299545325.41 is a zombie and is no longer with us

timstiles
Engager

I reinstalled splunk to a different volume and now I get this message when trying to search for any string. How can I fix this ?

Tags (1)
0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

are there any errors/warnings in $SPLUNK_HOME/var/run/splunk/dispatch/1299545325.41/search.log?

0 Karma

jrodman
Splunk Employee
Splunk Employee

This message is telling you the search is a zombie.

The specific line of code that makes this determination in splunkd is:

bool isZombie = !(isDone || ScopedAliveProcessToken::isAlive(ddir / ALIVE_TOKEN));

Ie, if it is not done, and not still alive, it is a zombie. In other words, this is a search that crashed (or somehow otherwise stopped talking to the outside world), but still left some stuff lying around (which will be cleaned up later). So the search artifacts exist in some state, and you can identify that the job once existed, but it probably isn't useful for information for making IT decisions etc about.

The token is typically a named pipe, so you could have a case where the named pipe failed to be created or opened (permissions, fd exhaustion) but most likely there was a crash.

Check the var/log/splunk directory for crashlogs, and if you're getting them, create a diag and work with support. (both of these are talked about here)

and eventually create a support case.

Ledion_Bitincka
Splunk Employee
Splunk Employee

@nick - nope, all crashlogs go to $SPLUNK_HOME/var/logs/

0 Karma

sideview
SplunkTrust
SplunkTrust

Wont the search crash logs be over in /var/run/dispatch/1299545325.41/ ? Or does it write crash logs out to the splunk's logs too?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...