Deployment Architecture

Why didn't peer indexer cluster?

TryingSplunk
Explorer

I have a stand-alone SH with 3 peer(non-clustered) indexers. I tried adding a 4th non-cluster indexer as a peer. 2 days later /opt/splunk was 100% full. Anyone have this happen? Is the data new data or old data that was copied to that indexer? I had to remove that indexer from the peer but now I don’t know what the data is on that 4th indexer. Help. New to Splunk obviously. 

Labels (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @TryingSplunk ,

I'm not speaking of Replication Factor that's a parameter for Clustered Indexers.

On Forwarders' outputs.conf: did you configured autoloadbalancing or not, for more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.4/Forwarding/Setuploadbalancingd

Ciao.

Giuseppe

View solution in original post

inventsekar
Ultra Champion

Hi @TryingSplunk ... Maybe one or two upvotes/likes please for the replies which helped you.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @TryingSplunk ,

as @inventsekar asked, you should better describe at first why you added a not clustered Indexer and if in this indexer you're sending all the data sources that you're sending also to the cluster or a subset.

Maybe you're sending both to cluster and not clustered IDX the same all data, this means that the not clustered IDX occupes much disk space that the cluster and probably this is the reason for the full disk.

Ciao.

Giuseppe

TryingSplunk
Explorer

Each indexer has the same indexes.conf file except the cold path is different. Which is why I assumed (badly) the data was the exact same data on each indexer.

0 Karma

inventsekar
Ultra Champion

Hi @TryingSplunk ...one question.. may i ask, why not into "indexer cluster" yet?!?!


when the 4th indexer /opt/splunk became full, we hope the first 3 were still working good, right

how do you manage the outputs.conf.. i mean, which logs you send to which indexer?

when you added the 4th indexer, did you accidently updated all UF's to send logs to 4th indexer alone(leaving the first 3 indexers idle)

TryingSplunk
Explorer

The indexers aren’t clustered because when I saw they were all peers I thought they were all getting the same data. Basically I’m new and misunderstood what a peer group was vs a clustered group. I did update the HF to include the 4th indexer. But I didn’t exclude the other 3 from the list.

0 Karma

TryingSplunk
Explorer

No I take that back. I didn’t make any changes to the HF but to the outputs.conf file to include the new indexer.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @TryingSplunk,

in addition you should check if you configured autoloadbalancing, in this case events are indexed once, otherwise data are indexed more times.

Then, are the first three indexed clustered or not?

Ciao.

Giuseppe

TryingSplunk
Explorer

No the first 3 indexers are not clustered. They are in a peer group. Looking at the indexes.conf file the repfactor = auto but not on the main ones like _internal, _audit.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @TryingSplunk ,

I'm not speaking of Replication Factor that's a parameter for Clustered Indexers.

On Forwarders' outputs.conf: did you configured autoloadbalancing or not, for more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.4/Forwarding/Setuploadbalancingd

Ciao.

Giuseppe

TryingSplunk
Explorer

The forwarder output.conf file was my issue. Because of that output.conf file I have a duplicate copy of my data. Once I clear that up then I’m going to tackle clustering the indexers. 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @TryingSplunk ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

TryingSplunk
Explorer

Well that just made this more confusing. Only one of the forwarders has an output.conf file.  The other ones do not. The forwarder that has the output.conf file is pointing to the indexer that’s full. That explains why when I brought this indexer online it immediately consumed so much. But that output.conf file was already sitting on that forwarder cause I didn’t even think about looking at forwarders when I added the additional indexer. Even stranger is why can I search the data with the indexer no longer in the peer group?  I’ve been going through db by db checking dates of indexes and searching and it’s all there. 

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...