Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About richardgosnay
richardgosnay
Explorer
Member since:
2 weeks ago
yesterday
Community Statistics
| Posts | 8 |
| Solutions | 0 |
| Karma Given | 2 |
| Karma Received | 0 |
| Member Since | 04-06-2021 |
Activity Feed
- Posted Restart order/procedure after updating server.conf with new master_uri & pass4SymmKey on Deployment Architecture. Monday
- Karma Re: Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) for scelikok. a week ago
- Karma Re: Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) for scelikok. a week ago
- Posted Re: Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) on Splunk Enterprise. a week ago
- Posted Re: Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) on Splunk Enterprise. a week ago
- Posted Re: Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) on Splunk Enterprise. a week ago
- Posted Re: Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) on Splunk Enterprise. a week ago
- Posted Re: Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) on Splunk Enterprise. a week ago
- Posted Re: Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) on Splunk Enterprise. a week ago
- Tagged Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) on Splunk Enterprise. a week ago
- Posted Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) on Splunk Enterprise. a week ago
- Tagged Cluster Index Bucket Stuck as "In Flight" - Roll, Resync and Delete Fails (Status=PendingDiscard) on Splunk Enterprise. a week ago
Topics I've Started
Monday
Hi Guys, I have: 1 x Search Node 1 x Master Node - 2 x Peer Nodes 1 x Deployment Node I've updated the master_uri & pass4SymmKey in the Search node and restarted it Splunk via the GUI, this worked fine and the license page is showing the new values. However, I am a bit reluctant to just change/restart the other nodes for fear of any bucket/replication issues. Am I ok to update the Master Node the same and perform a normal restart? Then update the Peer Nodes and perform a Rolling Restart? If not, what it is the best way to apply the conf changes and apply them? I did try searching the documentation but I got a bit lost. Thank you in advance.
... View more
Labels
- Labels:
-
indexer clustering
-
other
-
search head
a week ago
Thank you greatly, I will be performing the peer restarts in around 7 hours time. I'll let you know if it works and upvote accordingly.
... View more
a week ago
The bucket ID is the same, but the range at the beginning is not. Example Bucket ID: _audit~110~25359C10-2544-436D-893A-657C950D7863 Peer 1 Folder Name: rb_1614134403_1612300619_110_25359C10-2544-436D-893A-657C950D7863 Peer 2 Folder Name: db_1614134587_1612300619_110_25359C10-2544-436D-893A-657C950D7863 All neighbouring folder names match perfectly, it's only the buckets in question that don't match. If I remove the RB folder, will it get re-created with the correct DB equivalent?
... View more
a week ago
It seems, the only buckets affected are the replicated ones (rb instead of db). If I manually removed these before restarting the Cluster Master (and peers) will they just be re-created?
... View more
a week ago
The only errors in Splunk are the same as the ones in the splunkd.log file, you can see the snippet in the original post.
... View more
a week ago
If I manually locate the buckets in question, they don't have inflight- in the filename, they appear as normal buckets. But every time I try to run a fix up task like Roll, Resync or Delete, the log files states it is in flight (see previous log snippet). Should I try running the fix up tasks in maintenance mode?
... View more
a week ago
Hi, I'm currently running Splunk 7.3.0 and have 32 indexes running in a single cluster with 2 peers. Indexes are being replicated across both peers. Everything was working fine until we experienced a network blip 12 days ago, now I've noticed that the Replication Factor is not being met because there are some buckets from this time period which don't match, an average of about 3 buckets. I've tried to Roll, Resync and Delete these buckets via the GUI but each step fails. When I check splunkd.log, it appears as if Splunk is automatically trying to recover from these Fix Up tasks but it keeps reporting that the bucket is still in flight so can't. 04-06-2021 08:07:39.618 +0100 INFO CMSlave - truncate request bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bytes=0x0 current bid status=Complete 04-06-2021 08:07:39.618 +0100 INFO CMSlave - bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 Transitioning status from=Complete to=PendingDiscard for reason="schedule delete bucket" 04-06-2021 08:07:39.618 +0100 WARN CMSlave - event=scheduleDeleteBucket, bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bucket already in flight 04-06-2021 08:07:39.618 +0100 ERROR CMSlave - event=scheduleDeleteBucket, bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bucket already in flight 04-06-2021 08:07:39.618 +0100 INFO CMSlave - bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 Transitioning status from=PendingDiscard to=Complete for reason="failed to schedule delete bucket" 04-06-2021 08:07:39.618 +0100 ERROR ClusterSlaveBucketHandler - truncate bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bytes=0x0 earliest=0 latest=0 err='bucket already in flight' 04-06-2021 08:07:39.618 +0100 INFO CMSlave - truncate request bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bytes=0x0 current bid status=Complete 04-06-2021 08:07:39.619 +0100 INFO CMSlave - bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 Transitioning status from=Complete to=PendingDiscard for reason="schedule delete bucket" 04-06-2021 08:07:39.619 +0100 WARN CMSlave - event=scheduleDeleteBucket, bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bucket already in flight 04-06-2021 08:07:39.619 +0100 ERROR CMSlave - event=scheduleDeleteBucket, bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bucket already in flight 04-06-2021 08:07:39.619 +0100 INFO CMSlave - bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 Transitioning status from=PendingDiscard to=Complete for reason="failed to schedule delete bucket" 04-06-2021 08:07:39.619 +0100 ERROR ClusterSlaveBucketHandler - truncate bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bytes=0x0 earliest=0 latest=0 err='bucket already in flight' 04-06-2021 08:07:39.620 +0100 INFO CMSlave - Received resync bucket request for bid=bel1_qa_apps~19028~25359C10-2544-436D-893A-657C950D7863 bucketExists=1 04-06-2021 08:07:39.620 +0100 INFO CMSlave - Received resync bucket request for bid=bel1_qa_apps~19090~25359C10-2544-436D-893A-657C950D7863 bucketExists=1 Because of this, the Generation ID is also increasing quite rapidly. The status for all the buckets in question is stuck on 'PendingDiscard'. The same messages are appearing on the second node but with different bucket IDs. The same ID's keep repeating every few seconds on both peers. Should I restart each peer one at a time in hope that the bucket status is released and the fix up jobs can run as normal? Do I need to restart the cluster master? Any advice is appreciated. Thank you
... View more
Labels
- Labels:
-
administration
-
troubleshooting
