Getting Data In

Where do I start to look for Forwarder issues?

justin_deutsch
Explorer

I have a a number of light weight forwarders pointing to a single heavy forwarder point which in turn points to a single search head/indexer. The heavy forwarder has DB Connect installed.

Versions
Light Weight Forwarders: 6.1.0
Heavy Forwarder: 6.0.1
Search Head/Indexer: 6.0.1
(yes I know that we should upgrade the heavy forwader and search head/indexer)

Up until 31 Oct everything was working fine and then some events have stopped being forwarded to the indexer. For example, I can see that the database I am monitoring is generating the batch files on the forwarder and they are being processed in the logs on the server, but I can't see them when I search the index.

I would like to know is where I should start looking to see why the events aren't being forward. Any help would be appreciated.

Tags (2)
1 Solution

fabiocaldas
Contributor

You can start by reading splunkd.log files on your forwarders and indexer, it's can be found at $SPLUNK_HOME$/var/log/splunk folder. This log use be very helpful.

In my case I use a pair of applications to help me identify problems. The first app is s.o.s (Splunk on Splunk) installed on my search head and I have s.o.s add-on installed in all my Splunk Instance provinding an overall view about all machines. I also use another app to help me understand better my forwarders, it's called Forwarder Health.

View solution in original post

fabiocaldas
Contributor

You can start by reading splunkd.log files on your forwarders and indexer, it's can be found at $SPLUNK_HOME$/var/log/splunk folder. This log use be very helpful.

In my case I use a pair of applications to help me identify problems. The first app is s.o.s (Splunk on Splunk) installed on my search head and I have s.o.s add-on installed in all my Splunk Instance provinding an overall view about all machines. I also use another app to help me understand better my forwarders, it's called Forwarder Health.

Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...