Why Search Head become slow when can not connect to HF/IDX

[dillencehsu]

My environment is one Search Head -> one Heavy Forwerder -> 3 Indexers with Indexer Cluster.

Search Head become slow on Web UI after can not connect the Heavy Forwarder or Indexers.

I tried 2 scenarios,
(1) Search Head -> Heavy Forwarder -> Indexers (via SSL)
When I stop Heavy Forwarder for maintenance, the Search Head Web UI become very slow even hard to operate on Web UI and TailReader-0 become red until the Heavy Forwarder start.

(2) Search Head (directly to) -> Indexers (via SSL)
The same result with scenarios (1).

Why Splunk Search Head crashed after can not connect Heavy Forwarder or Indexer ?
When queue full just can not input data anymore, right ? What relate with splunkweb ?

英語だけではなく、
よろしければ、日本語で返事していただければ幸いです。
どうぞよろしくお願いいたします。

[harsmarvania57]

Why Search Head configured to search data from Heavy Forwarder ??

[dillencehsu]

I just tried different output targets, but the same result of my test.

[harsmarvania57]

Your search head need to configure to send data directly to Indexer, have a look at doc https://docs.splunk.com/Documentation/Splunk/8.0.2/DistSearch/Forwardsearchheaddata

To configure search head to search data from Indexer cluster, have a look at doc https://docs.splunk.com/Documentation/Splunk/8.0.2/Indexer/Enablethesearchhead

Have you configured your search head as given link above to forward the data and search data ?

[dillencehsu]

Yes, I configured.
My situation is not Search Head can not send data to Indexers.

When My Indexer can be connected, Search Head is well, when Indexers can not connected by Search Head, that will crashed (Web UI become slow even Web UI can not be access.)

I can understand input data will stop when output stop, why Web UI will be impact ?

[harsmarvania57]

Have you looked at crash logs in $SPLUNK_HOME/var/log/splunk/ ? At the time of crash any error in $SPLUNK_HOME/var/log/splunk/web_service.log ?

[dillencehsu]

Finally, I found the root cause is not related any .conf.
I copied worn ssl certificates for splunk-2-splunk forwarding.

Until I noticed and changed right self-signed certificates, Search Head is forwarding data to indexers well and have no any warn/error log about forwarding.

Thanks your reply and suggestion.

[harsmarvania57]

Its good that you found the problem and solved it, you can convert your comment to answer and accept it so that it will helpful for community member in future.