Unable to forward data into splunk

[PowderedSugar]

I am trying to implement a simple Splunk system on my local computer to learn a bit about how you set up forwards and get data into Splunk.  

I am running Splunk Enterprise on a CentOS 8 virtual machine, and I've installed a Universal Forwarder on the system that is running the virtual machine.  I've set up Splunk to receive data over port 9997, and have ensured that port 9997 is open and listening in CentOS.

On my main system I installed the Universal Forwarder and directed it to 192.168.0.21:9997 (my client is accessed at 192.168.0.21:8000).   

Outputs.conf:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 192.168.0.21:9997

[tcpout-server://192.168.0.21:9997]

I am not using a deployment server.

I'm using Bitdefender on my laptop and have made sure there's a rule in the firewall to allow traffic to 192.168.0.21:9997.  I've also reset the UF, Splunk Enterprise, and the VM running Splunk Enterprise.  

When I go in to Add Data > Forward, it still says "There are currently no forwarders configured as deployment clients to this instance."

I'm sure I'm just missing something in the setup steps, but I cannot figure out what it is.

----------------

Here are the main repeating messages from splunkd.log:

08-26-2021 16:21:40.575 -0800 INFO  AutoLoadBalancedConnectionStrategy [12416 TcpOutEloop] - Found currently active indexer. Connected to idx=192.168.0.21:9997, reuse=1.

08-26-2021 16:21:40.991 -0800 ERROR ExecProcessor [5456 ExecProcessor] - message from "D:\Cybersecurity\SplunkUniversalForwarder\bin\splunk-admon.exe" splunk-admon - GetLocalDN: Failed to get object 'LDAP://rootDSE': err='0x8007054b' - 'The specified domain either does not exist or could not be contacted.'

08-26-2021 16:21:40.991 -0800 ERROR ExecProcessor [5456 ExecProcessor] - message from "D:\Cybersecurity\SplunkUniversalForwarder\bin\splunk-admon.exe" splunk-admon - getBasePath: Unable to query local DN, restart and specify base path to monitor

08-26-2021 16:21:40.991 -0800 ERROR ExecProcessor [5456 ExecProcessor] - message from "D:\Cybersecurity\SplunkUniversalForwarder\bin\splunk-admon.exe" splunk-admon - SplunkADMon::configure: Failed to configure AD Monitor

[ephemeric]

It is not recommended to run a UF and indexer/search head on a single VM but for testing and learning it might work.

As the log shows: your UF has connected to the IDX.

That is correct: your UF is not configured as a deployment client as you are not using a deployment server. Nothing wrong here.

An example check for UF and IDX:

# UF.
$> splunk list forward-server
Active forwards:
	spl-idx-01.ephemeric.lan:9997

# IDX.
$> ss -ant | grep 9997
ESTAB      170697 0      10.10.50.206:9997               10.10.50.209:44208

[PowderedSugar]

@ephemeric @ Sorry, I must have mistyped something.  The indexer/search header is running on the VM, the UF is running on my laptop.  Now, the VM is running on my laptop as well, so if that's an issue please let me know.

[gcusello]

HI @PowderedSugar,

let me understand:

• the IP address of the Splunk Server on Virtual Machine is 192.168.0.21,
• the IP address of the target with Universal Forwarder is 192.168.0.21;

Is it correct?

If this is your configuration, please try to assign a different IP to the Splunk Server in the same class.

then you can check the connection using Telnet

Ciao.

Giuseppe

[PowderedSugar]

@gcusello : Here are my IPs

VM hosting Splunk Enterprise: 192.168.0.21

Laptop that has the UF on it: 192.168.0.26

The laptop with the UF is also the laptop that is running the VM through virtualbox.

I tried to connect over telnet but it wasn't able to.  I have added a firewall exception for port 23 on the VM, so I'm not sure why.  It's not showing up in the list of listening ports.  (8000 and 9997 are on the list)

[gcusello]

Hi @PowderedSugar,

as you can test with the telnet command that @ephemeric hinted, the port to open on Splunk Enterprise is 9997 if you used the default port.

The question is: did you enabled Splunk Enterprise on VM to receive logs from Forwarders?

if not, go in [Settings -- Forwarding and receiving -- Receiving] and enable it.

for more infos see at:

https://www.splunk.com/en_us/training/videos/getting-data-in-with-forwarders.html

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/Usingforwardingagents

https://docs.splunk.com/Documentation/Splunk/8.2.2/Forwarding/Aboutforwardingandreceivingdata 

Ciao.

Giuseppe

[ephemeric]

08-26-2021 16:21:40.575 -0800 INFO  AutoLoadBalancedConnectionStrategy [12416 TcpOutEloop] - Found currently active indexer. Connected to idx=192.168.0.21:9997, reuse=1.

It has connected.

[PowderedSugar]

@ephemeric  - 

Any thoughts as to why Splunk does not recognize that data is being forwarded to it?  I've made sure that 9997 is enabled as a receiving port in Splunk.  I go to Add Data > Forwarders, but I'm not seeing any potential forwarder sources listed to select from.  It still says "There are currently no forwarders configured as deployment clients to this instance."

That telnet command worked.  I was trying telnet 192.168.0.21:9997.  I'm still fairly new to this stuff.  

[ephemeric · Accepted solution]

Your UF has connected. By default a UF is not a deployment client, so you have no deployment clients as the GUI shows you.

Search 'index="_internal"' and you'll see your UF data there.

Do the `ss -ant | grep 9777` and you'll see your UF connected.

Check metrics.log on your indexer and you'll see UF data there too.

[PowderedSugar]

You appear to have been right about it sending data in already.  I am getting data from my laptop into the indexer.  Now to try to add a forwarder to my raspberry pi and see what happens.

I'm still not seeing anywhere to review the forwarder and see it coming in in Splunk.  Any chance you could point me to where that is?

Thanks for all the help!

[nyc_jason]

For the UF to collect data, it needs to have an inputs.conf configured to tell it what to collect. There is a default inputs.conf with the UF where you can enable things for it to collect, or you can install TAs (addons) from splunk base, which will come with their own inputs.conf. The outputs.conf tells the UF where to send the data to (you have that part set up fine). A Deployment server is able to distribute those TAs with inputs.conf (and other files) to the UFs, so you can centrally manage what they collect.

The method you are using to try and add data is for UFs that connect to your Deployment Server. Since you are not using the Deployment server, you will need to enable the inputs.conf on your UF directly. ie, install/configure the TAs/addons on the UF manually.  See here: https://docs.splunk.com/Documentation/Forwarder/8.2.2/Forwarder/Configuretheuniversalforwarder

the default inputs.conf that ships with the windows UF has lots of inputs setup to collect all sorts of data, but they are disabled by default. in the inputs.conf, is where they can be enabled.

[ephemeric]

$> telnet 192.168.0.21 9997