Spunk Forwarder troubleshooting

[lloydknight]

Here's the scenario:

UniversalForwarder1 already forwarding logs to Indexer1.
UniversalForwarder1's IP is 10.226.xx.xx and Indexer1's IP is 10.251.xx.xx
Connectivity:
Firewall is good. Can Telnet at port9997 from UniversalForwarder1 to Indexer1.
Splunkd logs:
Logs are good, no errors and whatsoever. Indexing OS logs from TA_nix_add-on.

UniversalForwarder1 to forward logs to Indexer2.
UniversalForwarder1's IP is 10.226.xx.xx and Indexer2's IP is 10.2226.xx.xx
Connectivity:
No need for Firewall as they're directly connected (p2p). Can Telnet at port9997 from UniversalForwarder1 to Indexer2. Traceroute has 2 hops only as expected.
Splunkd logs:
No internal logs to troubleshoot. How is that? Not Indexing OS logs from TA_nix_add-on even though UniversalForwarder1 is sending logs to Indexer1 and Indexer1 is indexing logs from it. No logs from Indexer2.

Anyone?

[nit123]

Check if the DNS is resolved when the forwarder sends data to indexer. Are there any unknown host error at the network level ?

More info shall help me address your problem.

[lloydknight]

For the Network level, ping, traceroute, and telnet were good. what other tests should I do here?

checked %SPLUNK/var/log/splunk/splunkd.log on the server with installed Forwarder, Forwarder is connected to the Indexer1 but no logs pertaining to Indexer2.

It's as if the error is indexer IP and port was not defined in outputs.conf but quadruple checked it already.

I want to provide more info but I'm stuck as there are no logs 😞

any recommendations?

[nit123]

Just to reiterate that you have done the following. Kindly confirm

1) Setup a Forwarder

To enable forwarding, navigate to Settings -> Forwarding & Receiving -> Configure Forwarding -> New & set IP address of the splunk instance to forward data to.

2) Setup a Indexer

All full Splunk Enterprise instances serve as indexers by default.

To forward remote data to an indexer, you use forwarders, which are Splunk Enterprise instances that receive data inputs and then consolidate and send the data to a Splunk Enterprise indexer.
To enable receiver at Indexer,

Navigate to Settings -> Forwarding & Receiving ->Configure Receiving -> New & add IP address of splunk stance that will forward data.

Have you followed the same steps ?

[lloydknight]

Hello, we are already on a production environment. Hundreds of Splunk UFs are already reporting to our Deployment client so yeah, already done with those steps.

Made some edits to make things more clear.

[nit123]

Alright. So are you indexing data on the forwarder as well or only forwarding data to indexer.

Without the logs having possible errors, we might not zero down to a root cause 😞