Getting Data In

Where do I start to look for Forwarder issues?

justin_deutsch
Explorer

I have a a number of light weight forwarders pointing to a single heavy forwarder point which in turn points to a single search head/indexer. The heavy forwarder has DB Connect installed.

Versions
Light Weight Forwarders: 6.1.0
Heavy Forwarder: 6.0.1
Search Head/Indexer: 6.0.1
(yes I know that we should upgrade the heavy forwader and search head/indexer)

Up until 31 Oct everything was working fine and then some events have stopped being forwarded to the indexer. For example, I can see that the database I am monitoring is generating the batch files on the forwarder and they are being processed in the logs on the server, but I can't see them when I search the index.

I would like to know is where I should start looking to see why the events aren't being forward. Any help would be appreciated.

Tags (2)
1 Solution

fabiocaldas
Contributor

You can start by reading splunkd.log files on your forwarders and indexer, it's can be found at $SPLUNK_HOME$/var/log/splunk folder. This log use be very helpful.

In my case I use a pair of applications to help me identify problems. The first app is s.o.s (Splunk on Splunk) installed on my search head and I have s.o.s add-on installed in all my Splunk Instance provinding an overall view about all machines. I also use another app to help me understand better my forwarders, it's called Forwarder Health.

View solution in original post

fabiocaldas
Contributor

You can start by reading splunkd.log files on your forwarders and indexer, it's can be found at $SPLUNK_HOME$/var/log/splunk folder. This log use be very helpful.

In my case I use a pair of applications to help me identify problems. The first app is s.o.s (Splunk on Splunk) installed on my search head and I have s.o.s add-on installed in all my Splunk Instance provinding an overall view about all machines. I also use another app to help me understand better my forwarders, it's called Forwarder Health.

Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...