topic Splunk UF not forwarding data. in Splunk Cloud Platform

Splunk UF not forwarding data.

wj742 — Mon, 03 Mar 2025 06:41:42 GMT

We have a UF installed on one of the windows servers, all the configurations seem fine, and the ports are also opened still the server is not forwarding the data to Splunk.

Re: Splunk UF not forwarding data.

kiran_panchavat — Mon, 03 Mar 2025 06:42:49 GMT

@wj742

Check the splunkd.log for more detailed information.

Re: Splunk UF not forwarding data.

kiran_panchavat — Mon, 03 Mar 2025 06:44:53 GMT

@wj742

Check Splunk Process: Ensure the Splunk Universal Forwarder (UF) process is running.
Verify Configuration Files: Ensure inputs.conf and outputs.conf are correctly configured. Here are sample configurations:

# inputs.conf
[monitor://C:\path\to\logs]
disabled = false
sourcetype = my_sourcetype

# outputs.conf
[tcpout:my_indexer]
server = indexer_hostname:9997

Ensure the Splunk user has the necessary permissions to read the log files being monitored.

Re: Splunk UF not forwarding data.

kiran_panchavat — Mon, 03 Mar 2025 06:52:43 GMT

@wj742

First, ensure the UF is actually running on the Windows server:

Open the Services panel (services.msc) and look for "SplunkForwarder"
Confirm it’s running. If it’s stopped, start it.
If it’s running, restart it to rule out a temporary glitch (right-click > Restart).

Check UF Logs for Errors

The UF logs can reveal why data isn’t being forwarded. On the Windows server:
Navigate to C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log
Open splunkd.log in a text editor and look for:
ERROR or WARN messages, especially around the time you restarted the service or when forwarding should have occurred.
Key phrases like TcpOutputProc (indicates connection issues to the indexer) or FileInputTracker (indicates issues reading monitored files).

Common issues to spot:

Connect to <indexer_IP>:<port> failed" – suggests a network or indexer problem.
"Paused the data flow" – indicates a forwarding block, often due to indexer issues

Validate Forwarding Configuration

Even if you think the configuration is fine, let’s double-check the UF’s outputs.conf:
Location: C:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf (or in an app directory like etc\apps\<app_name>\local\ if managed by a deployment server).

Example of a correct configuration:

[tcpout]
defaultGroup = my_indexers

[tcpout:my_indexers]
server = <indexer_IP>:9997

Things to verify:

The server line points to the correct indexer IP and port (typically 9997).
No typos in the IP or port.
disabled = false (or omitted, as false is default).
If changes are made, restart the UF: C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe restart.

Confirm Network Connectivity

Since you’ve said ports are open, let’s test connectivity explicitly: From the Windows server, open a Command Prompt and run:

telnet <indexer_IP> 9997 (replace with your indexer’s IP and port).

If it connects (blank screen), the connection is good. If it fails ("Connect failed"), there’s a network issue despite open ports.

Alternative: Use PowerShell:

Test-NetConnection -ComputerName <indexer_IP> -Port 9997

If it fails:

Double-check the firewall on the Windows server (outbound TCP 9997).
Check the indexer’s firewall (inbound TCP 9997).
Confirm with your network team that no intermediate devices (e.g., proxies, NATs) are blocking traffic.

Verify Indexer Receiving Configuration

The indexer must be configured to receive data. Ensure it’s set to listen on the expected port (e.g., 9997).
Validate Inputs Configuration
The UF needs to know what data to forward. Check inputs.conf:
Location: C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf (or an app directory).

Example for Windows Event Logs:

[WinEventLog://Application]
disabled = false
index = my_index

Re: Splunk UF not forwarding data.

wj742 — Mon, 03 Mar 2025 07:26:19 GMT

1. Splunk process is running on the server.

2. Configured the correct inputs under inputs.conf and outputs. conf

###### OS Logs ######
[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 300
index = wineventlog
renderXml=false

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 300
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist3 = EventCode="5447"
index = wineventlog
renderXml=false