Getting Data In

Spunk Forwarder troubleshooting

lloydknight
Builder

Here's the scenario:

UniversalForwarder1 already forwarding logs to Indexer1.
UniversalForwarder1's IP is 10.226.xx.xx and Indexer1's IP is 10.251.xx.xx
Connectivity:
Firewall is good. Can Telnet at port9997 from UniversalForwarder1 to Indexer1.
Splunkd logs:
Logs are good, no errors and whatsoever. Indexing OS logs from TA_nix_add-on.

UniversalForwarder1 to forward logs to Indexer2.
UniversalForwarder1's IP is 10.226.xx.xx and Indexer2's IP is 10.2226.xx.xx
Connectivity:
No need for Firewall as they're directly connected (p2p). Can Telnet at port9997 from UniversalForwarder1 to Indexer2. Traceroute has 2 hops only as expected.
Splunkd logs:
No internal logs to troubleshoot. How is that? Not Indexing OS logs from TA_nix_add-on even though UniversalForwarder1 is sending logs to Indexer1 and Indexer1 is indexing logs from it. No logs from Indexer2.

Anyone?

0 Karma

nit123
Path Finder

Check if the DNS is resolved when the forwarder sends data to indexer. Are there any unknown host error at the network level ?

More info shall help me address your problem.

0 Karma

lloydknight
Builder

For the Network level, ping, traceroute, and telnet were good. what other tests should I do here?

checked %SPLUNK/var/log/splunk/splunkd.log on the server with installed Forwarder, Forwarder is connected to the Indexer1 but no logs pertaining to Indexer2.

It's as if the error is indexer IP and port was not defined in outputs.conf but quadruple checked it already.

I want to provide more info but I'm stuck as there are no logs 😞

any recommendations?

0 Karma

nit123
Path Finder

Just to reiterate that you have done the following. Kindly confirm

1) Setup a Forwarder

To enable forwarding, navigate to Settings -> Forwarding & Receiving -> Configure Forwarding -> New & set IP address of the splunk instance to forward data to.

2) Setup a Indexer

All full Splunk Enterprise instances serve as indexers by default.

To forward remote data to an indexer, you use forwarders, which are Splunk Enterprise instances that receive data inputs and then consolidate and send the data to a Splunk Enterprise indexer.
To enable receiver at Indexer,

Navigate to Settings -> Forwarding & Receiving ->Configure Receiving -> New & add IP address of splunk stance that will forward data.

Have you followed the same steps ?

0 Karma

lloydknight
Builder

Hello, we are already on a production environment. Hundreds of Splunk UFs are already reporting to our Deployment client so yeah, already done with those steps.

Made some edits to make things more clear.

0 Karma

nit123
Path Finder

Alright. So are you indexing data on the forwarder as well or only forwarding data to indexer.

Without the logs having possible errors, we might not zero down to a root cause 😞

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...