Deployment Architecture

Get data from different Splunk Instances?

gyarici
Path Finder

Hi,

I have 3 different instances that are totally separate.

  1. First one is Standalone single SH Enterprise server
  2. Indexer Clustering Enterprise Servers
  3. In this 3rd instance, I want to create a kind of Distributed Search to get 2 separate data/search-result from first and second instances.

Which architecture and configuration is the best to collect data from mix of architectures?

  • Distributed Search?
  • Any API?
  • curl? -etc...

Thanks

Labels (2)
0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

Using Splunk to search other Splunk instances can be accomplished as noted via distributed search.. There are a few different approaches to this depending on the architecture you are connecting to and want to search..

1) Connecting a SH to an indexer cluster -- You need to connect the standalone SH to the cluster by joining the cluster via adding the Cluster Master (CM) as the search peer, or join the cluster via . This will enable the SH to perform standard search against the cluster. If you add individual indexers as search peers, you can also search this data but this isn't the correct way to search against a cluster though, so be cautious and follow the process listed above and here : https://docs.splunk.com/Documentation/Splunk/8.0.3/Indexer/Configureclusteredandnonclusteredsearch

2) Connecting to a "Standalone" Indexer. This process is different then joining a cluster. You have to add the Indexer as a Distributed Search Peer. For the initial setup, this will require that you have an admin user account on the Indexer to join it as a peer. Once joined successfully you should see successful status.

So be aware there are differences depending on what you are connecting to. E.g., a Cluster or a standalone indexer.

Regarding the 303 error you are getting, this is probably related to adding the clusters indexers as search peers and not joining the cluster properly. Make sure you go through the proper process as documented above.

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Using Splunk to search other Splunk instances can be accomplished as noted via distributed search.. There are a few different approaches to this depending on the architecture you are connecting to and want to search..

1) Connecting a SH to an indexer cluster -- You need to connect the standalone SH to the cluster by joining the cluster via adding the Cluster Master (CM) as the search peer, or join the cluster via . This will enable the SH to perform standard search against the cluster. If you add individual indexers as search peers, you can also search this data but this isn't the correct way to search against a cluster though, so be cautious and follow the process listed above and here : https://docs.splunk.com/Documentation/Splunk/8.0.3/Indexer/Configureclusteredandnonclusteredsearch

2) Connecting to a "Standalone" Indexer. This process is different then joining a cluster. You have to add the Indexer as a Distributed Search Peer. For the initial setup, this will require that you have an admin user account on the Indexer to join it as a peer. Once joined successfully you should see successful status.

So be aware there are differences depending on what you are connecting to. E.g., a Cluster or a standalone indexer.

Regarding the 303 error you are getting, this is probably related to adding the clusters indexers as search peers and not joining the cluster properly. Make sure you go through the proper process as documented above.

View solution in original post

0 Karma

gyarici
Path Finder

So, in this architecture, are standalone Splunk server able to connect both indexer cluster SH ( connect to indexer master too) and standonale other Splunk Server as Distributed Search at the same time?
You mentioned OR in your description.

Thanks

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Stand alone Splunk Enterprise instances (both acting as a search head, or indexer) can search both other standalone indexers and also indexing clusters. The main point to this is though, that the method for connecting to these are different.

At a high level, to connect to a stand alone (non-clustered) indexer, you simply add that indexer as a search peer. The process for joining a cluster is different however, since we have to connect to the Cluster Master (CM) role in order to search the data in that cluster correctly.

The number of peers and clusters a SH can search against is near unlimited. But there are architectural decisions to make when you have multiple deployments. This is mostly around bandwidth and network latency between the SH and peers you are searching. The Deploying and Architecting Splunk manual covers this more in detail ( https://doc.splunk.com)

0 Karma

gyarici
Path Finder

Thanks alot for the details. Now I am able to add entire cluster using standalone SH successfully.
2 indexers,1 Master,1 SH(Site 1) ---> Central SH(Site 2) <---Standalone SH (Site 3)

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Based on my understanding you have 3rd instance as standalone Search Head and want to search data from 1st standalone splunk instance and 2nd Indexer cluster servers. If that is the case then go through this doc https://docs.splunk.com/Documentation/Splunk/8.0.3/Indexer/Configureclusteredandnonclusteredsearch and configure your 3rd instance.

0 Karma

gyarici
Path Finder

Thanks for the answer. I checked this out and I have many errors to connect SH for indexer cluster from Single Server SH.

Check credentials/firewalls. Any experience on this?

ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error

ERROR DistributedBundleReplicationManager - Bundle Replication: Problem replicating config (bundle) to search peer ' xxxxxx:8089 ', HTTP response code 303 (HTTP/1.1 303 See Other) 

ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named xxxx  with uri=https://xxxxxx:8089.

WARN DistributedPeerManager - Cannot determine a latest common bundle, search may be blocked
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Which steps have you performed ?

0 Karma

gyarici
Path Finder

Connect to Indexer cluster SH via using standalone server. What I realize from @esix_splunk comment below is to do same thing for indexer master too. This could be the issue(?)

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

As @esix_splunk mentioned and if you follow document properly, you can see that to search data from Indexer Cluster you need to point your Search Head to Cluster Master (No need to add clustered indexer in distsearch.conf on SH). To search data from standalone instance you need to add that instance as search peer on SH(3rd Instance).

0 Karma