topic Re: How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server? in Deployment Architecture

How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server?

stephanedeck — Tue, 29 Sep 2020 23:07:51 GMT

Hi all, I did read and try numerous if not all the subject similar to mine.
I installed a Deployment Server on my Splunk Enterprise Server.

I followed the tutorial and made the "sendtoindexer" app following Splunk App for Windows Infrastructure 1.4 documentation. Everything works fine. I did put the "Splunk_TA_Windows" in the correct folders on Deployment Server.

Infact everything works perfectly, except that my Universal Forwarder on the Deployment client doesn't use the outputs.conf from the "sendtoindexer" app...

The outputs.conf file is in the folder

When I'am looking at the splunkd.log on the UF I do have this....

02-11-2019 15:16:04.947 +1100 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
02-11-2019 15:16:16.497 +1100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
02-11-2019 15:16:22.364 +1100 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
02-11-2019 15:16:28.497 +1100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

I admit the message "Please configure outputs.conf" is pretty obvious but can't solve my problem....

but when i troubleshot with "splunk btool outputs list --debug" there's no use of the file:

I did restart, uninstall/install multiple times the UF, but it never works, I can't see any logs in my Splunk Enterprise instance.

But when I just copy the outputs.conf file from
C:\Program Files\SplunkUniversalForwarder\etc\apps\sendtoindexer
to C:\Program Files\SplunkUniversalForwarder\etc\system\local
and restart my UF, everything works fine and the logs are sended to my splunk instance....so no network problems... and the debug command show the stanzas from the conf file.

So the conf file is OK....

I'am pretty lost right now...made so many tests...

Help please.

Re: How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server?

stephanedeck — Mon, 11 Feb 2019 04:53:51 GMT

Just for precisions, there's no problems with the outputs.conf because it works when in the system\local folder... just in case you'll ask 🙂

Re: How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server?

stephanedeck — Tue, 29 Sep 2020 23:07:53 GMT

other information, sorry I'am very tired...
you'll find bellow the logs from splunkd.log of the UF when it's not working:

I admit the message is pretty obvious, but unable to solve the problem....

Re: How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server?

vishaltaneja070 — Mon, 11 Feb 2019 07:08:49 GMT

@stephanedeck

If you want to send outputs.conf using app from deployment server then it should be on the correct folder.
please place the outputs.conf file in deployment server,in sendtoindexer\local folder.

And then try. It will work for you.
And also after making changes in deployment server then run the below command on deployment server:
splunk reload deploy-server

Re: How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server?

stephanedeck — Mon, 11 Feb 2019 08:55:32 GMT

Thanks for the reply vishaltaneja07011993 you must be right... I'll try it asap (wednesday in fact) and tell you. Then I'll hit myself hard, I'am so stupid... Thanks à lot

Re: How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server?

woodcock — Mon, 11 Feb 2019 10:02:25 GMT

You are probably logging firewall blocks. Check those logs for blocks to port "8089" and also to ALL of the IPs of your Deployment Server. See if you can telnet Your.DS.IP.HERE 8089 from your client. If you get a login prompt you are good. If not, you are blocked (possibly you have no route defined).

Re: How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server?

stephanedeck — Wed, 13 Feb 2019 22:33:23 GMT

vishaltaneja07011993 as anticipated You were rigth!
thanks a lot

Re: How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server?

vishaltaneja070 — Thu, 14 Feb 2019 13:28:43 GMT

No Problem 🙂
Good Luck:)