Why didn't peer indexer cluster?

[TryingSplunk]

I have a stand-alone SH with 3 peer(non-clustered) indexers. I tried adding a 4th non-cluster indexer as a peer. 2 days later /opt/splunk was 100% full. Anyone have this happen? Is the data new data or old data that was copied to that indexer? I had to remove that indexer from the peer but now I don’t know what the data is on that 4th indexer. Help. New to Splunk obviously. 

[inventsekar]

Hi @TryingSplunk ... Maybe one or two upvotes/likes please for the replies which helped you.

[gcusello]

Hi @TryingSplunk ,

as @inventsekar asked, you should better describe at first why you added a not clustered Indexer and if in this indexer you're sending all the data sources that you're sending also to the cluster or a subset.

Maybe you're sending both to cluster and not clustered IDX the same all data, this means that the not clustered IDX occupes much disk space that the cluster and probably this is the reason for the full disk.

Ciao.

Giuseppe

[TryingSplunk]

Each indexer has the same indexes.conf file except the cold path is different. Which is why I assumed (badly) the data was the exact same data on each indexer.

[inventsekar]

Hi @TryingSplunk ...one question.. may i ask, why not into "indexer cluster" yet?!?!

when the 4th indexer /opt/splunk became full, we hope the first 3 were still working good, right

how do you manage the outputs.conf.. i mean, which logs you send to which indexer?

when you added the 4th indexer, did you accidently updated all UF's to send logs to 4th indexer alone(leaving the first 3 indexers idle)

[TryingSplunk]

The indexers aren’t clustered because when I saw they were all peers I thought they were all getting the same data. Basically I’m new and misunderstood what a peer group was vs a clustered group. I did update the HF to include the 4th indexer. But I didn’t exclude the other 3 from the list.

[TryingSplunk]

No I take that back. I didn’t make any changes to the HF but to the outputs.conf file to include the new indexer.

[gcusello]

Hi @TryingSplunk,

in addition you should check if you configured autoloadbalancing, in this case events are indexed once, otherwise data are indexed more times.

Then, are the first three indexed clustered or not?

Ciao.

Giuseppe

[TryingSplunk]

No the first 3 indexers are not clustered. They are in a peer group. Looking at the indexes.conf file the repfactor = auto but not on the main ones like _internal, _audit.

[gcusello · Accepted solution]

Hi @TryingSplunk ,

I'm not speaking of Replication Factor that's a parameter for Clustered Indexers.

On Forwarders' outputs.conf: did you configured autoloadbalancing or not, for more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.4/Forwarding/Setuploadbalancingd

Ciao.

Giuseppe

[TryingSplunk]

The forwarder output.conf file was my issue. Because of that output.conf file I have a duplicate copy of my data. Once I clear that up then I’m going to tackle clustering the indexers. 

[gcusello]

Hi @TryingSplunk ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[TryingSplunk]

Well that just made this more confusing. Only one of the forwarders has an output.conf file.  The other ones do not. The forwarder that has the output.conf file is pointing to the indexer that’s full. That explains why when I brought this indexer online it immediately consumed so much. But that output.conf file was already sitting on that forwarder cause I didn’t even think about looking at forwarders when I added the additional indexer. Even stranger is why can I search the data with the indexer no longer in the peer group?  I’ve been going through db by db checking dates of indexes and searching and it’s all there.