Unable to see all indexers by default.

[CableB0y]

I've recently started a new job, of which one of my duties is to take over managing splunk from the previous administrator who left short notice. I've got SOME lab knowledge of splunk, and have been a user, but never an administrator so I'm not really sure if we're meeting best practice with this current architecture.

Our current design is 1 x (SearchHead/Indexer), and three search peers, all of which are configured within 'Distributed Search -> Search Peers'. The problem is that whenever I search any events, I only see events from distributed peers, and not from the indexer running on the search head. I'm able to verify that it is indeed indexing data, but I have to explicitly add the SPL 'splunk_server=*'. Is this due to my current configuration, or is it possible I'm missing something in the configs?

Appreciate any help anyone can provide!

[gcusello]

Hi @CableB0y ,

check if (as Splunk best practices say!) your indexer is configured to forward its logs to Indexers [Settings -- Forward and Receiving -- Forwarding].

If yes, check if in your Indexers there's the missing index, if not add it.

Ciao.

Giuseppe

[CableB0y]

Hey thanks for the reply!

I checked and it doesn't look like forwarding is configured. I'm assuming based on the configuration that this will send all logs including internal logs to our other Splunk indexers, and I'm wondering what constraints that would have on bandwidth since some indexers are geographically separated.

I was able to fix the issue temporarily by modifying distsearch.conf on the search head to include an entry for localhost, but again, this was because the GUI wouldn't accept the hostname of the searchhead, so I'm thinking this is a poor long term solution.

[isoutamo · Accepted solution]

Usually in distributed environment the same node is not SH and IDX at the same time. It’s against best practices. I think that until you fix it you need to use it also a search peer in distsearch.conf or always add splunk_server=* to your searches.

Here is some instructions which could help you:

https://docs.splunk.com/Documentation/Splunk/8.0.4/InheritedDeployment/Introduction
https://www.splunk.com/pdfs/technical-briefs/splunk-validated-architectures.pdf

r. Ismo

[CableB0y]

I think you're 100% correct. I made the change in the UI originally but was prevented from adding the hostname of the SH, so I ended up making the change on the SH distsearch.conf file last week, restarted, and it seems to be working correctly now. I think we'll likely discuss moving the searchhead off the IDX, but appreciate the assistance in the interim!