Deployment Architecture

Search member are not appearing in monitoring console

btshivanand
Path Finder

Hi All,

I am setting up splunk cluster environment. IN which i have 1 deployer and cluster master and 4 indexer and 3 search head.

after setting up cluster now i am setting monitoring console on deployer . Unfortunately i am not able to see the  search head mambers in destributed search. i was able to see all 4 indexers but not search heads.

Can you please suggest what was the issue.?

 

Please let me know any input required

Labels (1)
Tags (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Did you add the search heads as peers to the MC?  On the MC/Deployer, go to Settings->Distributed Search and click the Add New link.  Enter the information for a SH.  Repeat for the other SHs.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @btshivanand,

There are two checks:

  • as indicated by @richgalloway, check if you added all the SHs as peers to the MC,
  • then you have check if all the SHs are configured to forward their internal logs to the Indexers.

Ciao.

Giuseppe

0 Karma

btshivanand
Path Finder

Thanks for your reply... Can you please tell me how i need forward my internal logs to indexer...

 

i builded search head cluster and then i joined them master with below command.

 

./splunk edit cluster-config -mode searchhead -master_uri https://8089 -secret XXXX

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @btshivanand,

each Splunk server (not the Indexers obviously) should send its internal logs to the Indexers,

The way is the one described by @richgalloway.

Ciao.

Giuseppe

0 Karma

btshivanand
Path Finder

Thanks for you kind answer.I was trying set up the one of search head which is not clsuster to send logs to indexer.i was not at succesfull.i created app and i defined output.conf as below..Can you susggest me is there any suggestion

# Turn off indexing on the search head
[indexAndForward]
index = false

[tcpout]
defaultGroup = my_search_peers
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:my_search_peers]
server = X:9997, X:9997, X:9997, X:9997

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @btshivanand,

you can do the same thing by UI [Settings -- Forwarding and Receiving -- Forward].

Ciao.

Giuseppe

0 Karma

btshivanand
Path Finder

I am getting below error after adding all the indexers

 

The TCP output processor has paused the data flow. Forwarding to host_dest= inside output group default-autolb-group from host_src=XXXX has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. Learn more.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @btshivanand,

you have to forward the logs of all Splunk Servers except Indexers because they are already indexed.

Ciao.

Giuseppe

 

0 Karma

btshivanand
Path Finder

Thanks.. issue resolved...port 9997 was blocked from zone2 where our search head is running..

 

Thanks alot for the help

0 Karma

btshivanand
Path Finder

i See one more message.

 

Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make the SHs forwarder their logs by adding an outputs.conf file to their configuration, like you would do for a universal forwarder.

---
If this reply helps you, an upvote would be appreciated.

richgalloway
SplunkTrust
SplunkTrust

Did you add the search heads as peers to the MC?  On the MC/Deployer, go to Settings->Distributed Search and click the Add New link.  Enter the information for a SH.  Repeat for the other SHs.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

btshivanand
Path Finder

Thanks alot i was able to add search head into the monitoring console

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!