Getting Data In

Splunk offline timeouted but returns ERR_NOERR

huszti21
Explorer

Hey,

I have a question regarding timeouts and return codes when Splunk is shutting down a cluster peer on a Linux system.
I ran a script that issues a "splunk offline", waits for the command to return, and then starts the next action unless the previous command comes back with a non-zero return code.
If that happens, the script stops and asks for the user's input, to either abort, retry, skip, or continue.
We encountered a situation where the offlining ran into a timeout and the command returned with Splunk still being in the process of terminating.
However, the script started the next command (which then stopped the flow when it detected an inconsistency), indicating that we received a ERR_NOERR return code from Splunk.
Is that expected Splunk behaviour?

Short info about the environment:
Splunk 6.6.5 (build b119a2a8b0ad)
multisite Indexer-Cluster with 16 peers

Thanks in advance!

0 Karma

nickhills
Ultra Champion

Shutting down Splunk on can take a while if the box is performing lots of searches as it will wait for these to stop.
Clustered Indexers can also take a long time as they try to finalise operations before the process quits.

Instead of waiting for the return from the offline command, I would poll the output from ./splunk status instead

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...