Splunk Enterprise

Why is Splunk not receiving data from forwarders?

ankurborah
Path Finder

Splunk not receiving data from forwarders. Host os Windows Server 2012 R2.

1. Restart Splunk forwarder not working, getting some error message on CMD prompt.

2. Re-install Splunk forwarder, data start indexing for a few minutes and stopped again

3. Checked Splunk forwarder service, all the time it is running state 

Getting below error(smaple part of the error) when restart forwarder:

No spec file for: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\app.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\inputs.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\props.conf
No spec file for: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\app.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf
Invalid key in stanza [WinHostMon://Host OperatingSystem] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 172: showZeroValue (value: 1).
Did you mean 'source'?
Did you mean 'source type'?
Invalid key in stanza [WinHostMon://Host Processor] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 179: showZeroValue (value: 1).
Did you mean 'source'?

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

If you reinstall the forwarder and everything seems to be working fine, then it stops, it suggests that the initial state of the forwarder after installation is ok and then it's being "misconfigured" by an app deployed from the deployment server which contains erroneous settings within the deployed app.

Do other forwarder contained within the same serverclass behave the same way?

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

It seems that you have quite old Windows version. Have you check that your UF version is supported on that OS level?

Error messages said that you have some unknown options in inputs.conf. Have you check that your TA is supported on your UF version?

r. Ismo

0 Karma

ankurborah
Path Finder

It was working till yesterday. Also, we are  monitoring similar types of os for other hosts.  There is no upgrade or downgrade of the issue hosts in the last 2 months.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Something has changed here:

Invalid key in stanza [WinHostMon://Host OperatingSystem] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 172: showZeroValue (value: 1).

Based on naming of this TA, you should as from your local Accenture staff if they can see what was wrong in this installation. 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Was there any OS updates/patching or was node or UF service restarted? If so, then the change which has broken it can be done a long time ago and now it has affected after restart. Almost every time there have been some changes if things goes broken. No you just need to find what that change was.

0 Karma

ankurborah
Path Finder

Windows patch updates happened every month on 26th on all hosts(400+).  Only this host stopped reporting on 1 Jun 2022. Then tried with restart 5th Jun.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...