Deployment Architecture

Why Search Head become slow when can not connect to HF/IDX

dillencehsu
Path Finder

My environment is one Search Head -> one Heavy Forwerder -> 3 Indexers with Indexer Cluster.

Search Head become slow on Web UI after can not connect the Heavy Forwarder or Indexers.

I tried 2 scenarios,
(1) Search Head -> Heavy Forwarder -> Indexers (via SSL)
When I stop Heavy Forwarder for maintenance, the Search Head Web UI become very slow even hard to operate on Web UI and TailReader-0 become red until the Heavy Forwarder start.

(2) Search Head (directly to) -> Indexers (via SSL)
The same result with scenarios (1).

Why Splunk Search Head crashed after can not connect Heavy Forwarder or Indexer ?
When queue full just can not input data anymore, right ? What relate with splunkweb ?

英語だけではなく、
よろしければ、日本語で返事していただければ幸いです。
どうぞよろしくお願いいたします。

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Why Search Head configured to search data from Heavy Forwarder ??

0 Karma

dillencehsu
Path Finder

I just tried different output targets, but the same result of my test.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Your search head need to configure to send data directly to Indexer, have a look at doc https://docs.splunk.com/Documentation/Splunk/8.0.2/DistSearch/Forwardsearchheaddata

To configure search head to search data from Indexer cluster, have a look at doc https://docs.splunk.com/Documentation/Splunk/8.0.2/Indexer/Enablethesearchhead

Have you configured your search head as given link above to forward the data and search data ?

0 Karma

dillencehsu
Path Finder

Yes, I configured.
My situation is not Search Head can not send data to Indexers.

When My Indexer can be connected, Search Head is well, when Indexers can not connected by Search Head, that will crashed (Web UI become slow even Web UI can not be access.)

I can understand input data will stop when output stop, why Web UI will be impact ?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Have you looked at crash logs in $SPLUNK_HOME/var/log/splunk/ ? At the time of crash any error in $SPLUNK_HOME/var/log/splunk/web_service.log ?

0 Karma

dillencehsu
Path Finder

Finally, I found the root cause is not related any .conf.
I copied worn ssl certificates for splunk-2-splunk forwarding.

Until I noticed and changed right self-signed certificates, Search Head is forwarding data to indexers well and have no any warn/error log about forwarding.

Thanks your reply and suggestion.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Its good that you found the problem and solved it, you can convert your comment to answer and accept it so that it will helpful for community member in future.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...