Splunk Enterprise

Why is Splunk not receiving data from forwarders?

ankurborah
Explorer

Splunk not receiving data from forwarders. Host os Windows Server 2012 R2.

1. Restart Splunk forwarder not working, getting some error message on CMD prompt.

2. Re-install Splunk forwarder, data start indexing for a few minutes and stopped again

3. Checked Splunk forwarder service, all the time it is running state 

Getting below error(smaple part of the error) when restart forwarder:

No spec file for: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\app.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\inputs.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_hostservice360-windows_adc_win-x86-64_iis\local\props.conf
No spec file for: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\app.conf
Checking: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf
Invalid key in stanza [WinHostMon://Host OperatingSystem] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 172: showZeroValue (value: 1).
Did you mean 'source'?
Did you mean 'source type'?
Invalid key in stanza [WinHostMon://Host Processor] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 179: showZeroValue (value: 1).
Did you mean 'source'?

Labels (1)
0 Karma

PickleRick
Ultra Champion

If you reinstall the forwarder and everything seems to be working fine, then it stops, it suggests that the initial state of the forwarder after installation is ok and then it's being "misconfigured" by an app deployed from the deployment server which contains erroneous settings within the deployed app.

Do other forwarder contained within the same serverclass behave the same way?

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

It seems that you have quite old Windows version. Have you check that your UF version is supported on that OS level?

Error messages said that you have some unknown options in inputs.conf. Have you check that your TA is supported on your UF version?

r. Ismo

0 Karma

ankurborah
Explorer

It was working till yesterday. Also, we are  monitoring similar types of os for other hosts.  There is no upgrade or downgrade of the issue hosts in the last 2 months.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Something has changed here:

Invalid key in stanza [WinHostMon://Host OperatingSystem] in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk-TA-acn_infra360host_adc_win-x86-64\local\inputs.conf, line 172: showZeroValue (value: 1).

Based on naming of this TA, you should as from your local Accenture staff if they can see what was wrong in this installation. 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Was there any OS updates/patching or was node or UF service restarted? If so, then the change which has broken it can be done a long time ago and now it has affected after restart. Almost every time there have been some changes if things goes broken. No you just need to find what that change was.

0 Karma

ankurborah
Explorer

Windows patch updates happened every month on 26th on all hosts(400+).  Only this host stopped reporting on 1 Jun 2022. Then tried with restart 5th Jun.

0 Karma
Get Updates on the Splunk Community!

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...