UNIVERSAL FORWARDER not forwarding logs

[UncleHugo]

Hey there,

I am actually new to Splunk and at the moment I don't know what I a doing wrong.

So I have a vm1 which has Splunk Enterprise deployed in it and another vm2 with the Universal Forwarder deployed in it.

My aim is to forward logs from vm2 to the Splunk Enterprise in vm1, up till now I am not still able to do that. Checked agent Mgt still not seeing any device added or even on Add Data (Select Forwarders).

How do I resolve this?

What am I doing wrong? Thanks in anticipation.  

[PickleRick]

Ok. Regardless of possible problems, it's important to understand that you're talking about two different things.

One is data forwarding from the UF to the indexer (in your case - the all-in-one box vm1). 

Another thing is forwarder management - providing forwarders with a configuration bundle.

These two things work separately. You can use the deployment server functionality to configure your forwarder to forward data (and it's usually easier to do it this way in big environments than to configure each forwarder by hand) but that's it.

Neither forwarder management (or agent management as it's called in Splunk 10) is required to configure data forwarding nor the data forwarding is needed to manage a forwarder with a Deployment Server.

First and foremost - do you have network connectivity between your vms? What OS are you using on them? (windows forwarder lets you configure some parameters during installation)

[UncleHugo]

Both VMs are Windows.

Vm 1 with the Splunk Enterprise runs on Windows datacenter 2019

Vm 2 with the UF runs on windows 11

[isoutamo]

What you have already done? 
Probably your listener is working as you can telnet to it. But are you using plain TCP (probably yes)?

What you have defined on VM side? Have you define that outputs is sent to Splunk VM1?

Have you anything in splunk.log on Server or UF side?
Can you see UF's log in server's internal logs?

[UncleHugo]

Yes I have network connectivity between the two vms, I can ping each other and I had to even open port 9997 on the Vm 1 that has the Splunk Enterprise making me telnet on port 9997 from the Vm that has the UF.

[PickleRick]

OK. Now we're getting somewhere 😉

Poet 9997 is for data forwarding. Forwarders register in Deployment Server on 8089. Also you need to configure the client to use a DS - with Windows UF, if you installed it interactively, you had a dialog during installation letting you specify the DS address. If you pointed it to your Splunk Server, enabled the DS functionality on your Splunk server and opened the 8089 port, the forwarder should register itself with the DS.

So that's the reason why you're not seeing your UF as deployment client.

If you provided target indexer address during UF installation, you should be (if everything works OK) seeing forwarders internal log in the _internal index.

[ITWhisperer]

Have you checked that you have networking capability between your vms? For example, is Splunk listening on a port on a network that is accessible from the other vm (you can use telnet to attempt a connection from one to the other)?

Do you have any errors reported in the splunk logs on either vm?

[UncleHugo]

Yeah, there is connectivity between both vms I can ping and I can telnet on port 9997.

Also no error message gotten.

I think I am not doing something right, just don't know which but to my knowledge I have done all I can.

[thahir]

@UncleHugo 

Enable the Deployment server on the server on your splunk home bath under bin dir

splunk enable deploy-server

Splukn restart

This is the switch that makes Forwarder Management(Agent management) show up in the UI.

On each client execute the below command

set deploy-poll <YourDeploymentServerFQDN>:8089 -auth admin:<password>

then restart the service, basically it will create the deploymentclient.conf on the client machine and it will include the below details

[deployment-client]
[target-broker:deploymentServer]
targetUri = <YourDeploymentServerFQDN>:8089

you can refer the below URL for more details,

https://help.splunk.com/en/splunk-enterprise/administer/update-your-deployment/9.0/configure-the-dep...