Deployment Architecture

Unable to see all indexers by default.

CableB0y
Engager

I've recently started a new job, of which one of my duties is to take over managing splunk from the previous administrator who left short notice. I've got SOME lab knowledge of splunk, and have been a user, but never an administrator so I'm not really sure if we're meeting best practice with this current architecture.

Our current design is 1 x (SearchHead/Indexer), and three search peers, all of which are configured within 'Distributed Search -> Search Peers'. The problem is that whenever I search any events, I only see events from distributed peers, and not from the indexer running on the search head. I'm able to verify that it is indeed indexing data, but I have to explicitly add the SPL 'splunk_server=*'. Is this due to my current configuration, or is it possible I'm missing something in the configs?

Appreciate any help anyone can provide!

Labels (1)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Usually in distributed environment the same node is not SH and IDX at the same time. It’s against best practices. I think that until you fix it you need to use it also a search peer in distsearch.conf or always add splunk_server=* to your searches.

Here is some instructions which could help you:

https://docs.splunk.com/Documentation/Splunk/8.0.4/InheritedDeployment/Introduction
https://www.splunk.com/pdfs/technical-briefs/splunk-validated-architectures.pdf

r. Ismo

View solution in original post

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @CableB0y ,

check if (as Splunk best practices say!) your indexer is configured to forward its logs to Indexers [Settings -- Forward and Receiving -- Forwarding].

If yes, check if in your Indexers there's the missing index, if not add it.

Ciao.

Giuseppe

0 Karma

CableB0y
Engager

Hey thanks for the reply!

I checked and it doesn't look like forwarding is configured. I'm assuming based on the configuration that this will send all logs including internal logs to our other Splunk indexers, and I'm wondering what constraints that would have on bandwidth since some indexers are geographically separated.

I was able to fix the issue temporarily by modifying distsearch.conf on the search head to include an entry for localhost, but again, this was because the GUI wouldn't accept the hostname of the searchhead, so I'm thinking this is a poor long term solution.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Usually in distributed environment the same node is not SH and IDX at the same time. It’s against best practices. I think that until you fix it you need to use it also a search peer in distsearch.conf or always add splunk_server=* to your searches.

Here is some instructions which could help you:

https://docs.splunk.com/Documentation/Splunk/8.0.4/InheritedDeployment/Introduction
https://www.splunk.com/pdfs/technical-briefs/splunk-validated-architectures.pdf

r. Ismo

Tags (1)
0 Karma

CableB0y
Engager

I think you're 100% correct. I made the change in the UI originally but was prevented from adding the hostname of the SH, so I ended up making the change on the SH distsearch.conf file last week, restarted, and it seems to be working correctly now. I think we'll likely discuss moving the searchhead off the IDX, but appreciate the assistance in the interim!

0 Karma
Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...