Deployment Architecture
Highlighted

Can anyone help me understand "genid thats invalid or out of range" and "Proxy is in disconnect state" warning messages from my indexer cluster?

Motivator

We have an indexer which is going to be down for several days, so in preparation for that I performed these steps:

  • splunk offline on site1 indexer
  • Changed server.conf on the cluster master node from this

    • sitereplicationfactor = origin:1,site1:1,site2:1,total:2 sitesearchfactor = origin:1,site1:1,site2:1,total:2
  • To this

    • sitereplicationfactor = origin:1,total:1 sitesearchfactor = origin:1,total:1
  • Bounced cluster master

I thought that would prevent search & replication messages from cluttering splunkd.log while site1 was unavailable, but I am getting this combination of warnings about 1500 times an hour:

WARN  CMMessages - got genid thats invalid or out of range, setting to INVALID_GENID, jn=18446744073709551616.000000
WARN  CMSlave - handleHeartbeatDone: successful heartbeat. Forcing a re-add, Reason="Proxy is in disconnect state."

The system also seems to think that the indexes are not fully searchable, but I think that there is a primary bucket for everything the remaining site2.

Should I do anything about these warnings?

0 Karma
Highlighted

Re: Can anyone help me understand "genid thats invalid or out of range" and "Proxy is in disconnect state" warning messages from my indexer cluster?

Esteemed Legend

I would open a support case.

0 Karma
Highlighted

Re: Can anyone help me understand "genid thats invalid or out of range" and "Proxy is in disconnect state" warning messages from my indexer cluster?

Motivator

I just realized that I never updated this with what I figured out.

On the cluster master ( etc\system\local\server.conf) I had to change this:

available_sites = site1, site2

to this:

available_sites = site2

View solution in original post