Data not reaching Splunk Cloud after Migrating to new Universal Forwarders

[Pete_]

Hello,

I am having issues getting data into Splunk Cloud with two new Universal forwarders.

I have two existing Universal Forwarders that are working just fine, but I am migrating these to new servers. Same Universal Forwarder version on both the old and new servers (9.4.3)

I have the Universal Forwader software installed on both the new Linux servers. I copied the inputs.conf and outputs.conf files from the old servers. I also installed splunkclouduf.spl that I downloaded from my Splunk Cloud instance.

The usage for these forwarders is limited to syslog messages only. I receive syslog messages from other devices on port 514 of the Universal Forwarders (UDP and TCP allowed) and those messages forward to Splunk Cloud. Pretty simple setup.

I have confirmed that traffic is being received on the servers on port 514 using tcpdump. However, none of that traffic is reaching Splunk Cloud.

I can see the new forwarders in the Splunk Cloud Monitoring Console under Forwarders->Versions and Forwarders->Instance.

But no data is being received from the new forwarders.

Below are my inputs.conf and outputs.conf files from one of the new servers. As you can see, very simple setup and outputs.conf is doing nothing. Again, these were copied from my old working servers exactly, except for the hostname on the new forwarders.
----------------------------------------
inputs.conf 

[default]
host = NHC-NETSplunkForwarder

[tcp://514]
acceptFrom = *
connection_host=ip
index=nhcnetwork
sourcetype=NETWORK
disabled=0

[udp://514]
acceptFrom = *
connection_host=ip
index=nhcnetwork
sourcetype=NETWORK

----------------------------------------
outputs.conf (sanitized)

#This breaks stuff. The credentials package provides what is needed here. Leave commented out.
#[tcpout]
#defaultGroup = splunkcloud,default-autolb-group

#[tcpout:default-autolb-group]
#server = XXXXXXX.splunkcloud.com:9997
#disabled = false

#[tcpout-server://XXXXXXX.splunkcloud.com:9997]

Do I need to do something in Splunk Cloud to allow these new forwarders to send data? I don't know how splunkclouduf.spl works so I don't know a way to monitor output traffic from the Universal Forwarder. Any suggestions or tips are appreciated.

Thanks,

-Pete

[livehybrid]

Hi @Pete_ 

The splunkclouduf.spl app configures secure forwarding to Splunk Cloud; you should not need to modify outputs.conf directly, also, because you're able to see the new forwarders in the Cloud Monitoring Console (CMC) we know that the outputs are established and the new UFs can reach Splunk Cloud.

The testing you've done shows the 514 syslog feed arriving at the box, however is Splunk listening on that port? If you run the following can you see that splunkd is listening to the port?

sudo netstat -tulnp | grep 514

Are there any logs in $SPLUNK_HOME/var/log/splunk/splunkd.log about binding port 514, any errors etc?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Pete_]

@livehybrid Again, forgive me if you get repeated replies from me. My replies are not showing after I post them. I'm brand new to the community so maybe I'm missing something silly.

To answer your questions,

sudo netstat -tulnp | grep 514

this returns nothing

However, plenty of errors in splunkd.log

root@NHC-NETSplunkForwarder:/opt/splunkforwarder/var/log/splunk# cat splunkd.log | grep "514"
06-25-2025 19:24:20.190 +0000 INFO  TcpInputConfig [59254 TcpListener] - IPv4 port 514 is reserved for raw input
06-25-2025 19:24:20.190 +0000 INFO  TcpInputConfig [59254 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7
06-25-2025 19:24:20.190 +0000 ERROR TcpInputProc [59254 TcpListener] - Could not bind to port IPv4 port 514: Permission denied
06-25-2025 19:26:21.991 +0000 INFO  TcpInputConfig [59507 TcpListener] - IPv4 port 514 is reserved for raw input
06-25-2025 19:26:21.991 +0000 INFO  TcpInputConfig [59507 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7
06-25-2025 19:26:21.992 +0000 ERROR TcpInputProc [59507 TcpListener] - Could not bind to port IPv4 port 514: Permission denied
06-25-2025 21:18:16.827 +0000 INFO  TcpInputConfig [60127 TcpListener] - IPv4 port 514 is reserved for raw input
06-25-2025 21:18:16.827 +0000 INFO  TcpInputConfig [60127 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7
06-25-2025 21:18:16.828 +0000 ERROR TcpInputProc [60127 TcpListener] - Could not bind to port IPv4 port 514: Permission denied
06-26-2025 01:38:09.514 +0000 INFO  AutoLoadBalancedConnectionStrategy [60145 TcpOutEloop] - Connected to idx=34.201.206.231:9997:0, pset=0, reuse=0. using ACK. autoBatch=1
06-26-2025 14:41:49.984 +0000 INFO  TcpInputConfig [63678 TcpListener] - IPv4 port 514 is reserved for raw input
06-26-2025 14:41:49.984 +0000 INFO  TcpInputConfig [63678 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7
06-26-2025 14:41:49.984 +0000 ERROR TcpInputProc [63678 TcpListener] - Could not bind to port IPv4 port 514: Permission denied

[richgalloway]

Those error messages are saying Splunk does not have permission to use port 514.  All ports <1024 are "privileged" and require special permission to access.  Running Splunk as root will solve that, but I highly discourage that.

The recommended practice is to send syslog data to a dedicated syslog receiver (syslog-ng, for example), have it write the data to disk, and have a UF monitor those disk files.  You also can use Splunk Connect 4 Syslog (SC4S) to send the data directly to Splunk.

[Pete_]

We will be installing Splunk Connect 4 Syslog soon. But I haven't got there yet. That will be more involved.

We previously tried running syslog-ng on the server and monitoring the file, but everything came into splunk cloud from the same host in Splunk Cloud. It was a mess.

When I installed the Universal Forwarder on the new servers, I created new user splunkfwd to run it, just like the instructions said. Can I simply change the permissions for user splunkfwd? At this point I don't really care if it runs with root privileges. what would the needed permissions for user splunkfwd to overcome this?

Thanks,

-Pete 

[richgalloway]

To allow the UF access to port 514, try this

setcap 'cap_net_bind_service=+ep' /path/to/uf

[Pete_]

setcap 'cap_net_bind_service=+ep' /opt/splunkforwarder/bin/splunk

I just tried this, still seeing the same issue.

I also had my system admin move user splunkfwd (this user runs splunk) into the sudo group 

still seeing the same errors in splunkd.log

06-26-2025 18:46:46.515 +0000 INFO  TcpInputConfig [921 TcpListener] - IPv4 port 514 is reserved for raw input
06-26-2025 18:46:46.515 +0000 INFO  TcpInputConfig [921 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7
06-26-2025 18:46:46.515 +0000 ERROR TcpInputProc [921 TcpListener] - Could not bind to port IPv4 port 514: Permission denied
06-26-2025 19:27:32.285 +0000 INFO  TcpInputConfig [1554 TcpListener] - IPv4 port 514 is reserved for raw input
06-26-2025 19:27:32.286 +0000 INFO  TcpInputConfig [1554 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7
06-26-2025 19:27:32.286 +0000 ERROR TcpInputProc [1554 TcpListener] - Could not bind to port IPv4 port 514: Permission denied

[Pete_]

Talked to my sysadmin, we decided to use port 1035 instead of port 514. not getting the socket errors in splunkd.log anymore, but still not seeing the messages from the UF in Splunk Cloud.  

root@NHC-NETSplunkForwarder:/opt/splunkforwarder/var/log/splunk# cat splunkd.log | grep "1035"
06-26-2025 20:05:00.017 +0000 INFO  TcpInputConfig [1851 TcpListener] - IPv4 port 1035 is reserved for raw input
06-26-2025 20:05:00.017 +0000 INFO  TcpInputConfig [1851 TcpListener] - IPv4 port 1035 will negotiate s2s protocol level 7
06-26-2025 20:05:00.017 +0000 INFO  TcpInputProc [1851 TcpListener] - Creating raw Acceptor for IPv4 port 1035 with Non-SSL
06-26-2025 20:25:30.471 +0000 WARN  AutoLoadBalancedConnectionStrategy [1869 TcpOutEloop] - Possible duplication of events with channel=source::udp:1035|host::10.12.2.149|NETWORK|, streamId=1989559377486376685, offset=6 on host=3.213.185.213:9997 connid 0