topic Splunk Forwarder issue in Getting Data In

Splunk Forwarder issue

saotaigiri — Fri, 14 Aug 2020 17:28:34 GMT

A forwarder which was working before has stopped for up to a month now. After checking, it is confirmed that the forwarder is correctly configured, the service is running and reporting to the indexer, however there is no data showing on the indexer side or when a search is run on splunk for that forwarder. Can someone please help me in the right direction?

Re: Splunk Forwarder issue

isoutamo — Fri, 14 Aug 2020 17:49:05 GMT

If you have MC installed then you could look from forwarder monitoring that it is there. If not installed then just look from index=_intenal host=<your forwarder>.

Is your forwarder UF or HF?
And which kind of inputs there are configured?
Was there e.g. OS update or something else before sending stops?
What you found from forwarders logs ?

r. Ismo

Re: Splunk Forwarder issue

thambisetty — Fri, 14 Aug 2020 19:25:11 GMT

Can you check the servername in system/local/server.conf and host in system/local/inputs.conf

The host and servername from Splunk config should match with server hostname.

and also take the guid from $SPLUNK_HOME/etc/instance.cfg and check in Search head to see how many forwarders are reporting with the GUID.

and also identify hostname associated with guid and check if its matching with server hostname.

Re: Splunk Forwarder issue

isoutamo — Fri, 14 Aug 2020 21:25:26 GMT

It’s best practices that those three names are same. But those two inside splunk must not to be a server’s hostname. Time by time (e.g. AWS when hostname changes “all” time) is much better to fix it as splunk instance name instead of physical hos/node name.

Re: Splunk Forwarder issue

burwell — Sat, 15 Aug 2020 05:33:58 GMT

I had those happen today. Restarting the forwarder fixed the problem.