Why am I not seeing Forwarders?

[MeeksFamily06]

Any help is appreciated

OK, I installed splunk on a docker instance, 

docker run -d --name Splunk --restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock -p 8000:8000 -p 8089:8089 -p 9997:9997 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=SUPER-SECRET" splunk/splunk:latest

Then I went to settings, forwarding and receiving, Receive data, Configure receiving and made sure Liston On Port 9997 was enabled
Added a new Username and new password

Then I went to an ubuntu 22.04 I think and ran (ChatGPT aided in some of this)

* sudo su
* useradd -m splunk
* groupadd splunk (Which if memory serves it said group already existed)
* export SPLUNK_HOME="/opt/splunkforwarder"
* mkdir $SPLUNK_HOME
* Then I cd'd into the splunk home directory
* chown -R splunk:splunk $SPLUNK_HOME
* wget -O splunkforwarder-9.0.5-Not Sure if these were account specific so removed them-linux-2.6-amd64.deb "https://download.splunk.com/products/universalforwarder/releases/9.0.5/linux/splunkforwarder-9.0.5-Not Sure if these were account specific so removed them-linux-2.6-amd64.deb"
* dpkg -i /path/to/splunkforwarder_package_name.deb
* chown -R splunk:splunk /opt/splunkforwarder
* sudo -u splunk /opt/splunkforwarder/bin/splunk add forward-server My-IP-Address-To-Docker:9997 -auth New-Username:New-Password

* That then made me agree and enter the username and password I created for Splunk in Docker

* sudo -u splunk /opt/splunkforwarder/bin/splunk set deploy-poll IP-to-Docker:8089
* sudo -u splunk /opt/splunkforwarder/bin/splunk restart

* Then I go to settings, Add Data, Forward and I see There are currently no forwarders configured as deployment clients to this instance.
* Also if I go to Forwarder management I see The forwarder management UI distributes deployment apps to Splunk clients. No clients or apps are currently available on this deployment server.

What am I doing wrong?

[gcusello · Accepted solution]

Hi @MeeksFamily06,

to send logs to a Splunk instance, you have to configure the outputs.conf in your Universal Forwarder or use a CLI command as 

./splunk add forward-server <host name or ip address>:<listening port

for more infos see

https://docs.splunk.com/Documentation/Forwarder/9.0.4/Forwarder/Configuretheuniversalforwarder

But this isn't sufficient to see the UF as a deployment client.

You have to configure the deploymentclient.conf file or run a CLI command:

splunk set deploy-poll <IP_address/hostname>:<management_port>
splunk restart

for more infos see:

https://docs.splunk.com/Documentation/Splunk/latest/Updating/Configuredeploymentclients

Ciao.

Giuseppe

[wangyu]

In fact, from this document "https://docs.splunk.com/Documentation/Forwarder/9.2.1/Forwarder/Consolidatedatafrommultiplehosts", I did not find that the second step needs to be executed.

[gcusello]

Hi @MeeksFamily06 ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[MeeksFamily06]

I had not originally ran 

sudo -u splunk /opt/splunkforwarder/bin/splunk set deploy-poll IP-to-Docker:8000 

BUT I just went back and ran that. Then ran 

sudo -u splunk /opt/splunkforwarder/bin/splunk restart

It didn't show up, then rebooted both entire machines just for good measure and still didn't see it

[isoutamo]

IP address:port for DS should be IP-to-Docker:8089 if you are using normal port. 8089 is REST port, 8000 for GUI and 9997 for receiving data.

[MeeksFamily06]

Ok, I deleted and reran my docker instance with (To also open port 8089)

docker run -d --name Splunk --restart unless-stopped  -v /var/run/docker.sock:/var/run/docker.sock -p 8000:8000 -p 8089:8089 -p 9997:9997 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=SUPER-SECRET" splunk/splunk:latest

Then I ran on the ubuntu server

*sudo -u splunk /opt/splunkforwarder/bin/splunk set deploy-poll IP-to-Docker-Server:8089
*sudo -u splunk /opt/splunkforwarder/bin/splunk restart

Still did not see it, then I rebooted everything for good measure and still do not see it

[isoutamo]

Hi

Have you start your UF? You just said that you add DS address, but didn't mention if you (re)start UF?

Are those one same network subnet and are you sure that there are no FW blocking that connection?

btw. When you use UF don't use same password for it than you are using on Your Splunk Server. UF's password should be different and even UF's internal account name could be different than normal admin.

You also should have some base app for your environment where you have defined your outputs.conf etc. for all UFs. Usually I have own app which contains also that DS connection part. I never use that "splunk add forward-server...." as it put its configuration under /opt/splunkforwarders/etc/system/local and then I cannot update it from DS side!

r. Ismo 

[MeeksFamily06]

I am sorry, I did run 

sudo -u splunk /opt/splunkforwarder/bin/splunk restart

and sometimes rebooted the entire system just for good measure. 

Also tbh I thought the first username and password I was entering was the login info for the receiving splunk instance, it was until later that I discovered it was the login info for the forwarder

The two computers are on the same router

The base app is the docker instance I believe