Getting Data In

Splunk offline timeouted but returns ERR_NOERR



I have a question regarding timeouts and return codes when Splunk is shutting down a cluster peer on a Linux system.
I ran a script that issues a "splunk offline", waits for the command to return, and then starts the next action unless the previous command comes back with a non-zero return code.
If that happens, the script stops and asks for the user's input, to either abort, retry, skip, or continue.
We encountered a situation where the offlining ran into a timeout and the command returned with Splunk still being in the process of terminating.
However, the script started the next command (which then stopped the flow when it detected an inconsistency), indicating that we received a ERR_NOERR return code from Splunk.
Is that expected Splunk behaviour?

Short info about the environment:
Splunk 6.6.5 (build b119a2a8b0ad)
multisite Indexer-Cluster with 16 peers

Thanks in advance!

0 Karma

Ultra Champion

Shutting down Splunk on can take a while if the box is performing lots of searches as it will wait for these to stop.
Clustered Indexers can also take a long time as they try to finalise operations before the process quits.

Instead of waiting for the return from the offline command, I would poll the output from ./splunk status instead

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...