*NIX app causing Splunkd to take forever restarting

[mundus]

We're running Splunk 4.2.1 build 98164 on Ubuntu 2.6.32-32-server. When the *NIX app is installed, Splunk takes several minutes to shutdown and the indexes come up as needing repair each time. If I remove the app, Splunk restarts quickly and without any index problems.

What's the best way to figure out what the problem is with the *NIX app?

Thx.

Craig

[dwaddle]

The nix app is heavily dependent on scripted inputs. These scripted input jobs **could* be running long and Splunk is waiting for them all to finish. I know the default netstat.sh does not give the "-n" option to netstat, which causes it to do name resolution (which can take a while if there's a lot of varying names to resolve).

When you are waiting on Splunk to shut down, try running something like this to see if Splunk is waiting on hanging out processes.

ps -ef | grep splunkd   # this should list two splunkd processes, one a child of the other.
# the child is the interesting one
while [ 1 ]; do pgrep -P <child_pid> | xargs -r ps -fp; done

This for loop is expensive, because it is busy waiting. But it'll give you a hint at least if Splunk is waiting on scripted inputs to finish.