Getting Data In

Where do I start to look for Forwarder issues?

justin_deutsch
Explorer

I have a a number of light weight forwarders pointing to a single heavy forwarder point which in turn points to a single search head/indexer. The heavy forwarder has DB Connect installed.

Versions
Light Weight Forwarders: 6.1.0
Heavy Forwarder: 6.0.1
Search Head/Indexer: 6.0.1
(yes I know that we should upgrade the heavy forwader and search head/indexer)

Up until 31 Oct everything was working fine and then some events have stopped being forwarded to the indexer. For example, I can see that the database I am monitoring is generating the batch files on the forwarder and they are being processed in the logs on the server, but I can't see them when I search the index.

I would like to know is where I should start looking to see why the events aren't being forward. Any help would be appreciated.

Tags (2)
1 Solution

fabiocaldas
Contributor

You can start by reading splunkd.log files on your forwarders and indexer, it's can be found at $SPLUNK_HOME$/var/log/splunk folder. This log use be very helpful.

In my case I use a pair of applications to help me identify problems. The first app is s.o.s (Splunk on Splunk) installed on my search head and I have s.o.s add-on installed in all my Splunk Instance provinding an overall view about all machines. I also use another app to help me understand better my forwarders, it's called Forwarder Health.

View solution in original post

fabiocaldas
Contributor

You can start by reading splunkd.log files on your forwarders and indexer, it's can be found at $SPLUNK_HOME$/var/log/splunk folder. This log use be very helpful.

In my case I use a pair of applications to help me identify problems. The first app is s.o.s (Splunk on Splunk) installed on my search head and I have s.o.s add-on installed in all my Splunk Instance provinding an overall view about all machines. I also use another app to help me understand better my forwarders, it's called Forwarder Health.

Get Updates on the Splunk Community!

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...