topic Re: Splunk not receiving data from forwarders in Splunk Enterprise

Why is Splunk not receiving data from forwarders?

ankurborah — Mon, 13 Jun 2022 19:57:05 GMT

Splunk not receiving data from forwarders. Host os Windows Server 2012 R2.

1. Restart Splunk forwarder not working, getting some error message on CMD prompt.

2. Re-install Splunk forwarder, data start indexing for a few minutes and stopped again

3. Checked Splunk forwarder service, all the time it is running state

Getting below error(smaple part of the error) when restart forwarder:

No spec file for: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\app.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\inputs.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\props.conf
No spec file for: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\app.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf
Invalid key in stanza [WinHostMon://Host OperatingSystem] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 172: showZeroValue (value: 1).
Did you mean 'source'?
Did you mean 'source type'?
Invalid key in stanza [WinHostMon://Host Processor] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 179: showZeroValue (value: 1).
Did you mean 'source'?

Re: Splunk not receiving data from forwarders

isoutamo — Mon, 13 Jun 2022 05:24:11 GMT

It seems that you have quite old Windows version. Have you check that your UF version is supported on that OS level?

Error messages said that you have some unknown options in inputs.conf. Have you check that your TA is supported on your UF version?

r. Ismo

Re: Splunk not receiving data from forwarders

ankurborah — Mon, 13 Jun 2022 05:27:47 GMT

It was working till yesterday. Also, we are monitoring similar types of os for other hosts. There is no upgrade or downgrade of the issue hosts in the last 2 months.

Re: Splunk not receiving data from forwarders

isoutamo — Mon, 13 Jun 2022 05:30:59 GMT

Was there any OS updates/patching or was node or UF service restarted? If so, then the change which has broken it can be done a long time ago and now it has affected after restart. Almost every time there have been some changes if things goes broken. No you just need to find what that change was.

Re: Splunk not receiving data from forwarders

ankurborah — Mon, 13 Jun 2022 07:26:09 GMT

Windows patch updates happened every month on 26th on all hosts(400+). Only this host stopped reporting on 1 Jun 2022. Then tried with restart 5th Jun.

Re: Splunk not receiving data from forwarders

isoutamo — Mon, 13 Jun 2022 13:34:19 GMT

Something has changed here:

Invalid key in stanza [WinHostMon://Host OperatingSystem] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 172: showZeroValue (value: 1).

Based on naming of this TA, you should as from your local Accenture staff if they can see what was wrong in this installation.

Re: Splunk not receiving data from forwarders

PickleRick — Mon, 13 Jun 2022 14:27:07 GMT

If you reinstall the forwarder and everything seems to be working fine, then it stops, it suggests that the initial state of the forwarder after installation is ok and then it's being "misconfigured" by an app deployed from the deployment server which contains erroneous settings within the deployed app.

Do other forwarder contained within the same serverclass behave the same way?