- Find Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Splunk Offline command - running for hours
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've opened a support ticket but hoping someone may have seen this. I have an indexer cluster with two indexers and a cluster master and I'm upgrading all of them from 6.4.3 to 6.4.6.
CM was upgraded and placed into maintenance mode. Indexer 1 was taken offline (by using "splunk offline"), upgraded and rebooted.
On Indexer 2, issued a "splunk offline" command, and it's still running 5 hours later. The machine isn't locked - the status "dots" keep filling up the command window.
Has anyone encountered this, or is anyone aware of a way to check the actual offline status and possibly close the window? I was following along with the upgrade procedure, but can't find any mention of this situation anywhere.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk support did contact me this morning. We weren't able to determine an exact cause of this behavior, but did find that stopping the Splunkd process caused it to stop hanging. Specifically, the command prompt window where I had run "Splunk offline" displayed a message indicating that primaries had been reassigned and it was complete. I set the splunk service to start manually, rebooted the server and then installed the Splunk 6.4.6 update. It appears to be working now - it rejoined the cluster and I have not seen any signs that there is a problem.
*** I'll accept this as an answer with a caveat. I recommend contacting Splunk support in this situation, as they may identify something in the splunkd.log that points to a root cause, or may indicate that you should not terminate the process like I did ***
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk support did contact me this morning. We weren't able to determine an exact cause of this behavior, but did find that stopping the Splunkd process caused it to stop hanging. Specifically, the command prompt window where I had run "Splunk offline" displayed a message indicating that primaries had been reassigned and it was complete. I set the splunk service to start manually, rebooted the server and then installed the Splunk 6.4.6 update. It appears to be working now - it rejoined the cluster and I have not seen any signs that there is a problem.
*** I'll accept this as an answer with a caveat. I recommend contacting Splunk support in this situation, as they may identify something in the splunkd.log that points to a root cause, or may indicate that you should not terminate the process like I did ***
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That's great to hear, thank you for the update. I will convert your last comment to an answer. If you could accept it, so the question shows as resolved for others that may run into the same situation, that'd be great. Thanks!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What OS are you running on?
Any error messages in the cluster master log?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Windows Server 2012 for the indexers. Windows Server 2012 R2 for the cluster master.
Looking in splunkd.log on the CM, nothing that seems out of place (to me, at least). I see error messages about regex statements hitting a match limit (I use regex to blacklist some events), some warnings about cooked connections, and one of my search heads which is currently offline.
Most of the log contains INFO events pertaining to CMBucket - event=isFixupComplete
Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...
Monitoring Amazon Elastic Kubernetes Service (EKS)
Cloud Platform & Enterprise: Classic Dashboard Export Feature Deprecation
