Deployment Architecture

How to check Splunk components connectivity in a distributed environment?

namrithadeepak
Path Finder

Hi,

How do I check the connectivity between:

(All of them are unix)

  • Search Head -> Indexer
  • Indexer -> Forwarder
  • Deployment server -> Deployment Client

What string should I look for in logs?
Since they are all unix machines, can I check the connectivity using a command or any other way?

Thanks,
Deepak

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

There are a few ways to check connectivity between these elements.

Search Head to Indexer:

Telnet on port 8089 between the SH <> IDX

You can also check the Search Peer status from the Search Head by navigating to Settings > Distributed Search > Search Peers. It will depict the IDX statuses.

Forwarder to Indexer:

Telnet from fwd to IDX on port 9997 (or whatever forward port you set in outputs.conf)

From the forwarder you can also grep $SPLUNK_HOME/var/log/splunk/splunkd.log for TcpOutputProc ( tail -100 splunkd.log | grep TcpOutputProcto check if you are connecting to the indexers

From the indexer perspective you can search index=_internal sourcetype=splunkd tcpin_connectionsand confirm you see your forwarder

DS to Deployment Client:

Telnet on port 8089 from DS to Client and vice versa.
Check _internal logs for index=_internal sourcetype=splunkd deploymentClient to confirm phone home.

- MattyMo

View solution in original post

mattymo
Splunk Employee
Splunk Employee

There are a few ways to check connectivity between these elements.

Search Head to Indexer:

Telnet on port 8089 between the SH <> IDX

You can also check the Search Peer status from the Search Head by navigating to Settings > Distributed Search > Search Peers. It will depict the IDX statuses.

Forwarder to Indexer:

Telnet from fwd to IDX on port 9997 (or whatever forward port you set in outputs.conf)

From the forwarder you can also grep $SPLUNK_HOME/var/log/splunk/splunkd.log for TcpOutputProc ( tail -100 splunkd.log | grep TcpOutputProcto check if you are connecting to the indexers

From the indexer perspective you can search index=_internal sourcetype=splunkd tcpin_connectionsand confirm you see your forwarder

DS to Deployment Client:

Telnet on port 8089 from DS to Client and vice versa.
Check _internal logs for index=_internal sourcetype=splunkd deploymentClient to confirm phone home.

- MattyMo
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...