Getting Data In

Splunk Offline command - running for hours

baf879
Path Finder

I've opened a support ticket but hoping someone may have seen this. I have an indexer cluster with two indexers and a cluster master and I'm upgrading all of them from 6.4.3 to 6.4.6.

CM was upgraded and placed into maintenance mode. Indexer 1 was taken offline (by using "splunk offline"), upgraded and rebooted.

On Indexer 2, issued a "splunk offline" command, and it's still running 5 hours later. The machine isn't locked - the status "dots" keep filling up the command window.

Has anyone encountered this, or is anyone aware of a way to check the actual offline status and possibly close the window? I was following along with the upgrade procedure, but can't find any mention of this situation anywhere.

0 Karma
1 Solution

baf879
Path Finder

Splunk support did contact me this morning. We weren't able to determine an exact cause of this behavior, but did find that stopping the Splunkd process caused it to stop hanging. Specifically, the command prompt window where I had run "Splunk offline" displayed a message indicating that primaries had been reassigned and it was complete. I set the splunk service to start manually, rebooted the server and then installed the Splunk 6.4.6 update. It appears to be working now - it rejoined the cluster and I have not seen any signs that there is a problem.

*** I'll accept this as an answer with a caveat. I recommend contacting Splunk support in this situation, as they may identify something in the splunkd.log that points to a root cause, or may indicate that you should not terminate the process like I did ***

View solution in original post

baf879
Path Finder

Splunk support did contact me this morning. We weren't able to determine an exact cause of this behavior, but did find that stopping the Splunkd process caused it to stop hanging. Specifically, the command prompt window where I had run "Splunk offline" displayed a message indicating that primaries had been reassigned and it was complete. I set the splunk service to start manually, rebooted the server and then installed the Splunk 6.4.6 update. It appears to be working now - it rejoined the cluster and I have not seen any signs that there is a problem.

*** I'll accept this as an answer with a caveat. I recommend contacting Splunk support in this situation, as they may identify something in the splunkd.log that points to a root cause, or may indicate that you should not terminate the process like I did ***

s2_splunk
Splunk Employee
Splunk Employee

That's great to hear, thank you for the update. I will convert your last comment to an answer. If you could accept it, so the question shows as resolved for others that may run into the same situation, that'd be great. Thanks!

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

What OS are you running on?
Any error messages in the cluster master log?

0 Karma

baf879
Path Finder

Windows Server 2012 for the indexers. Windows Server 2012 R2 for the cluster master.

Looking in splunkd.log on the CM, nothing that seems out of place (to me, at least). I see error messages about regex statements hitting a match limit (I use regex to blacklist some events), some warnings about cooked connections, and one of my search heads which is currently offline.

Most of the log contains INFO events pertaining to CMBucket - event=isFixupComplete

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...