How to check Splunk components connectivity in a distributed environment?

[namrithadeepak]

Hi,

How do I check the connectivity between:

(All of them are unix)

• Search Head -> Indexer
• Indexer -> Forwarder
• Deployment server -> Deployment Client

What string should I look for in logs?
Since they are all unix machines, can I check the connectivity using a command or any other way?

Thanks,
Deepak

[mattymo · Accepted solution]

There are a few ways to check connectivity between these elements.

Search Head to Indexer:

Telnet on port 8089 between the SH <> IDX

You can also check the Search Peer status from the Search Head by navigating to Settings > Distributed Search > Search Peers. It will depict the IDX statuses.

Forwarder to Indexer:

Telnet from fwd to IDX on port 9997 (or whatever forward port you set in outputs.conf)

From the forwarder you can also grep $SPLUNK_HOME/var/log/splunk/splunkd.log for TcpOutputProc ( tail -100 splunkd.log | grep TcpOutputProcto check if you are connecting to the indexers

From the indexer perspective you can search index=_internal sourcetype=splunkd tcpin_connectionsand confirm you see your forwarder

DS to Deployment Client:

Telnet on port 8089 from DS to Client and vice versa.
Check _internal logs for index=_internal sourcetype=splunkd deploymentClient to confirm phone home.