Why is Splunk not receiving data from forwarders?

[ankurborah]

Splunk not receiving data from forwarders. Host os Windows Server 2012 R2.

1. Restart Splunk forwarder not working, getting some error message on CMD prompt.

2. Re-install Splunk forwarder, data start indexing for a few minutes and stopped again

3. Checked Splunk forwarder service, all the time it is running state 

Getting below error(smaple part of the error) when restart forwarder:

No spec file for: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\app.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\inputs.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\props.conf
No spec file for: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\app.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf
Invalid key in stanza [WinHostMon://Host OperatingSystem] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 172: showZeroValue (value: 1).
Did you mean 'source'?
Did you mean 'source type'?
Invalid key in stanza [WinHostMon://Host Processor] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 179: showZeroValue (value: 1).
Did you mean 'source'?

[PickleRick]

If you reinstall the forwarder and everything seems to be working fine, then it stops, it suggests that the initial state of the forwarder after installation is ok and then it's being "misconfigured" by an app deployed from the deployment server which contains erroneous settings within the deployed app.

Do other forwarder contained within the same serverclass behave the same way?

[isoutamo]

Hi

It seems that you have quite old Windows version. Have you check that your UF version is supported on that OS level?

Error messages said that you have some unknown options in inputs.conf. Have you check that your TA is supported on your UF version?

r. Ismo

[ankurborah]

It was working till yesterday. Also, we are  monitoring similar types of os for other hosts.  There is no upgrade or downgrade of the issue hosts in the last 2 months.

[isoutamo]

Something has changed here:

Invalid key in stanza [WinHostMon://Host OperatingSystem] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 172: showZeroValue (value: 1).

Based on naming of this TA, you should as from your local Accenture staff if they can see what was wrong in this installation. 

[isoutamo]

Was there any OS updates/patching or was node or UF service restarted? If so, then the change which has broken it can be done a long time ago and now it has affected after restart. Almost every time there have been some changes if things goes broken. No you just need to find what that change was.

[ankurborah]

Windows patch updates happened every month on 26th on all hosts(400+).  Only this host stopped reporting on 1 Jun 2022. Then tried with restart 5th Jun.