How to troubleshoot why a Universal Forwarder is not sending data to the Deployment Server?

[stephanedeck]

Hi all, I did read and try numerous if not all the subject similar to mine.
I installed a Deployment Server on my Splunk Enterprise Server.

I followed the tutorial and made the "sendtoindexer" app following Splunk App for Windows Infrastructure 1.4 documentation. Everything works fine. I did put the "Splunk_TA_Windows" in the correct folders on Deployment Server.

Infact everything works perfectly, except that my Universal Forwarder on the Deployment client doesn't use the outputs.conf from the "sendtoindexer" app...

The outputs.conf file is in the folder

When I'am looking at the splunkd.log on the UF I do have this....

02-11-2019 15:16:04.947 +1100 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
02-11-2019 15:16:16.497 +1100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
02-11-2019 15:16:22.364 +1100 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
02-11-2019 15:16:28.497 +1100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

I admit the message "Please configure outputs.conf" is pretty obvious but can't solve my problem....

but when i troubleshot with "splunk btool outputs list --debug" there's no use of the file:

I did restart, uninstall/install multiple times the UF, but it never works, I can't see any logs in my Splunk Enterprise instance.

But when I just copy the outputs.conf file from
C:\Program Files\SplunkUniversalForwarder\etc\apps\sendtoindexer
to C:\Program Files\SplunkUniversalForwarder\etc\system\local
and restart my UF, everything works fine and the logs are sended to my splunk instance....so no network problems... and the debug command show the stanzas from the conf file.

So the conf file is OK....

I'am pretty lost right now...made so many tests...

Help please.

[woodcock]

You are probably logging firewall blocks. Check those logs for blocks to port "8089" and also to ALL of the IPs of your Deployment Server. See if you can telnet Your.DS.IP.HERE 8089 from your client. If you get a login prompt you are good. If not, you are blocked (possibly you have no route defined).

[stephanedeck]

Thanks for the reply vishaltaneja07011993 you must be right... I'll try it asap (wednesday in fact) and tell you. Then I'll hit myself hard, I'am so stupid... Thanks à lot

[vishaltaneja070 · Accepted solution]

@stephanedeck

If you want to send outputs.conf using app from deployment server then it should be on the correct folder.
please place the outputs.conf file in deployment server,in sendtoindexer\local folder.

And then try. It will work for you.
And also after making changes in deployment server then run the below command on deployment server:
splunk reload deploy-server

[stephanedeck]

vishaltaneja07011993 as anticipated You were rigth!
thanks a lot

[vishaltaneja070]

No Problem 🙂
Good Luck:)

[stephanedeck]

Just for precisions, there's no problems with the outputs.conf because it works when in the system\local folder... just in case you'll ask 🙂

[stephanedeck]

other information, sorry I'am very tired...
you'll find bellow the logs from splunkd.log of the UF when it's not working:

02-11-2019 15:16:04.947 +1100 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
02-11-2019 15:16:16.497 +1100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
02-11-2019 15:16:22.364 +1100 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
02-11-2019 15:16:28.497 +1100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

I admit the message is pretty obvious, but unable to solve the problem....