Deployment Architecture

How to check Splunk components connectivity in a distributed environment?

namrithadeepak
Path Finder

Hi,

How do I check the connectivity between:

(All of them are unix)

  • Search Head -> Indexer
  • Indexer -> Forwarder
  • Deployment server -> Deployment Client

What string should I look for in logs?
Since they are all unix machines, can I check the connectivity using a command or any other way?

Thanks,
Deepak

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

There are a few ways to check connectivity between these elements.

Search Head to Indexer:

Telnet on port 8089 between the SH <> IDX

You can also check the Search Peer status from the Search Head by navigating to Settings > Distributed Search > Search Peers. It will depict the IDX statuses.

Forwarder to Indexer:

Telnet from fwd to IDX on port 9997 (or whatever forward port you set in outputs.conf)

From the forwarder you can also grep $SPLUNK_HOME/var/log/splunk/splunkd.log for TcpOutputProc ( tail -100 splunkd.log | grep TcpOutputProcto check if you are connecting to the indexers

From the indexer perspective you can search index=_internal sourcetype=splunkd tcpin_connectionsand confirm you see your forwarder

DS to Deployment Client:

Telnet on port 8089 from DS to Client and vice versa.
Check _internal logs for index=_internal sourcetype=splunkd deploymentClient to confirm phone home.

- MattyMo

View solution in original post

mattymo
Splunk Employee
Splunk Employee

There are a few ways to check connectivity between these elements.

Search Head to Indexer:

Telnet on port 8089 between the SH <> IDX

You can also check the Search Peer status from the Search Head by navigating to Settings > Distributed Search > Search Peers. It will depict the IDX statuses.

Forwarder to Indexer:

Telnet from fwd to IDX on port 9997 (or whatever forward port you set in outputs.conf)

From the forwarder you can also grep $SPLUNK_HOME/var/log/splunk/splunkd.log for TcpOutputProc ( tail -100 splunkd.log | grep TcpOutputProcto check if you are connecting to the indexers

From the indexer perspective you can search index=_internal sourcetype=splunkd tcpin_connectionsand confirm you see your forwarder

DS to Deployment Client:

Telnet on port 8089 from DS to Client and vice versa.
Check _internal logs for index=_internal sourcetype=splunkd deploymentClient to confirm phone home.

- MattyMo
Get Updates on the Splunk Community!

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...