topic Re: Spunk Forwarder troubleshooting in Getting Data In

Spunk Forwarder troubleshooting

lloydknight — Tue, 29 Sep 2020 14:26:24 GMT

Here's the scenario:

UniversalForwarder1 already forwarding logs to Indexer1.
UniversalForwarder1's IP is 10.226.xx.xx and Indexer1's IP is 10.251.xx.xx
Connectivity:
Firewall is good. Can Telnet at port9997 from UniversalForwarder1 to Indexer1.
Splunkd logs:
Logs are good, no errors and whatsoever. Indexing OS logs from TA_nix_add-on.

UniversalForwarder1 to forward logs to Indexer2.
UniversalForwarder1's IP is 10.226.xx.xx and Indexer2's IP is 10.2226.xx.xx
Connectivity:
No need for Firewall as they're directly connected (p2p). Can Telnet at port9997 from UniversalForwarder1 to Indexer2. Traceroute has 2 hops only as expected.
Splunkd logs:
No internal logs to troubleshoot. How is that? Not Indexing OS logs from TA_nix_add-on even though UniversalForwarder1 is sending logs to Indexer1 and Indexer1 is indexing logs from it. No logs from Indexer2.

Anyone?

Re: Spunk Forwarder troubleshooting

nit123 — Tue, 13 Jun 2017 07:43:07 GMT

Check if the DNS is resolved when the forwarder sends data to indexer. Are there any unknown host error at the network level ?

More info shall help me address your problem.

Re: Spunk Forwarder troubleshooting

lloydknight — Tue, 13 Jun 2017 08:58:59 GMT

For the Network level, ping, traceroute, and telnet were good. what other tests should I do here?

checked %SPLUNK/var/log/splunk/splunkd.log on the server with installed Forwarder, Forwarder is connected to the Indexer1 but no logs pertaining to Indexer2.

It's as if the error is indexer IP and port was not defined in outputs.conf but quadruple checked it already.

I want to provide more info but I'm stuck as there are no logs 😞

any recommendations?

Re: Spunk Forwarder troubleshooting

nit123 — Tue, 13 Jun 2017 12:03:54 GMT

Just to reiterate that you have done the following. Kindly confirm

1) Setup a Forwarder

To enable forwarding, navigate to Settings -> Forwarding & Receiving -> Configure Forwarding -> New & set IP address of the splunk instance to forward data to.

2) Setup a Indexer

All full Splunk Enterprise instances serve as indexers by default.

To forward remote data to an indexer, you use forwarders, which are Splunk Enterprise instances that receive data inputs and then consolidate and send the data to a Splunk Enterprise indexer.
To enable receiver at Indexer,

Navigate to Settings -> Forwarding & Receiving ->Configure Receiving -> New & add IP address of splunk stance that will forward data.

Have you followed the same steps ?

Re: Spunk Forwarder troubleshooting

lloydknight — Tue, 13 Jun 2017 13:32:38 GMT

Hello, we are already on a production environment. Hundreds of Splunk UFs are already reporting to our Deployment client so yeah, already done with those steps.

Made some edits to make things more clear.

Re: Spunk Forwarder troubleshooting

nit123 — Wed, 14 Jun 2017 05:42:20 GMT

Alright. So are you indexing data on the forwarder as well or only forwarding data to indexer.

Without the logs having possible errors, we might not zero down to a root cause 😞