Deployment Architecture

Why Search Head become slow when can not connect to HF/IDX

dillencehsu
Path Finder

My environment is one Search Head -> one Heavy Forwerder -> 3 Indexers with Indexer Cluster.

Search Head become slow on Web UI after can not connect the Heavy Forwarder or Indexers.

I tried 2 scenarios,
(1) Search Head -> Heavy Forwarder -> Indexers (via SSL)
When I stop Heavy Forwarder for maintenance, the Search Head Web UI become very slow even hard to operate on Web UI and TailReader-0 become red until the Heavy Forwarder start.

(2) Search Head (directly to) -> Indexers (via SSL)
The same result with scenarios (1).

Why Splunk Search Head crashed after can not connect Heavy Forwarder or Indexer ?
When queue full just can not input data anymore, right ? What relate with splunkweb ?

英語だけではなく、
よろしければ、日本語で返事していただければ幸いです。
どうぞよろしくお願いいたします。

0 Karma

harsmarvania57
Ultra Champion

Why Search Head configured to search data from Heavy Forwarder ??

0 Karma

dillencehsu
Path Finder

I just tried different output targets, but the same result of my test.

0 Karma

harsmarvania57
Ultra Champion

Your search head need to configure to send data directly to Indexer, have a look at doc https://docs.splunk.com/Documentation/Splunk/8.0.2/DistSearch/Forwardsearchheaddata

To configure search head to search data from Indexer cluster, have a look at doc https://docs.splunk.com/Documentation/Splunk/8.0.2/Indexer/Enablethesearchhead

Have you configured your search head as given link above to forward the data and search data ?

0 Karma

dillencehsu
Path Finder

Yes, I configured.
My situation is not Search Head can not send data to Indexers.

When My Indexer can be connected, Search Head is well, when Indexers can not connected by Search Head, that will crashed (Web UI become slow even Web UI can not be access.)

I can understand input data will stop when output stop, why Web UI will be impact ?

0 Karma

harsmarvania57
Ultra Champion

Have you looked at crash logs in $SPLUNK_HOME/var/log/splunk/ ? At the time of crash any error in $SPLUNK_HOME/var/log/splunk/web_service.log ?

0 Karma

dillencehsu
Path Finder

Finally, I found the root cause is not related any .conf.
I copied worn ssl certificates for splunk-2-splunk forwarding.

Until I noticed and changed right self-signed certificates, Search Head is forwarding data to indexers well and have no any warn/error log about forwarding.

Thanks your reply and suggestion.

0 Karma

harsmarvania57
Ultra Champion

Its good that you found the problem and solved it, you can convert your comment to answer and accept it so that it will helpful for community member in future.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...