Deployment Architecture

splunk clustered index upgrade to version 6

shervinfernando
Explorer

I upgraded my clustered index and after the upgrade /opt/splunk/bin/splunk show cluster-bundle-status shows all peer nodes with the status Detention.

Rolling restart of the peer nodes didn't fix this issue - can you let me know how to fix it

1 Solution

asetiawan
Explorer

I experienced similar issue when my indexer is out of disk space.

Splunk 6 seems to require min 5GB free space by default. So, if you have less than 5GB space on version 5.x and upgraded to version 6, your indexer will stop receiving logs and see warning messages like:


skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

You can still search indexed logs, but the indexer status on the cluster master's clustering dashboard will be "Detention"

View solution in original post

Ellen
Splunk Employee
Splunk Employee

shervinfernando
Explorer

Fixed it by adding following to server.conf file on the indexer reporting this error

[diskUsage]
minFreeSpace = 2000

0 Karma

ChrisG
Splunk Employee
Splunk Employee
0 Karma

asetiawan
Explorer

I experienced similar issue when my indexer is out of disk space.

Splunk 6 seems to require min 5GB free space by default. So, if you have less than 5GB space on version 5.x and upgraded to version 6, your indexer will stop receiving logs and see warning messages like:


skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

You can still search indexed logs, but the indexer status on the cluster master's clustering dashboard will be "Detention"

View solution in original post

shervinfernando
Explorer

Fixed it by adding following to server.conf file on the indexer reporting this error

[diskUsage]
minFreeSpace = 2000

0 Karma