Splunk UF not forwarding data.

[wj742]

We have a UF installed on one of the windows servers, all the configurations seem fine, and the ports are also opened still the server is not forwarding the data to Splunk.

[kiran_panchavat · Accepted solution]

@wj742 

First, ensure the UF is actually running on the Windows server:

• Open the Services panel (services.msc) and look for "SplunkForwarder"
• Confirm it’s running. If it’s stopped, start it.
• If it’s running, restart it to rule out a temporary glitch (right-click > Restart).

Check UF Logs for Errors

• The UF logs can reveal why data isn’t being forwarded. On the Windows server:
• Navigate to C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log
• Open splunkd.log in a text editor and look for:
• ERROR or WARN messages, especially around the time you restarted the service or when forwarding should have occurred.
• Key phrases like TcpOutputProc (indicates connection issues to the indexer) or FileInputTracker (indicates issues reading monitored files).

Common issues to spot:

• Connect to <indexer_IP>:<port> failed" – suggests a network or indexer problem.
• "Paused the data flow" – indicates a forwarding block, often due to indexer issues

Validate Forwarding Configuration

• Even if you think the configuration is fine, let’s double-check the UF’s outputs.conf:
• Location: C:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf (or in an app directory like etc\apps\<app_name>\local\ if managed by a deployment server).

Example of a correct configuration:

[tcpout]
defaultGroup = my_indexers

[tcpout:my_indexers]
server = <indexer_IP>:9997

Things to verify:

• The server line points to the correct indexer IP and port (typically 9997).
• No typos in the IP or port.
• disabled = false (or omitted, as false is default).
• If changes are made, restart the UF: C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe restart.

Confirm Network Connectivity

Since you’ve said ports are open, let’s test connectivity explicitly: From the Windows server, open a Command Prompt and run:

telnet <indexer_IP> 9997 (replace with your indexer’s IP and port).

If it connects (blank screen), the connection is good. If it fails ("Connect failed"), there’s a network issue despite open ports.

Alternative: Use PowerShell:

Test-NetConnection -ComputerName <indexer_IP> -Port 9997

If it fails:

• Double-check the firewall on the Windows server (outbound TCP 9997).
• Check the indexer’s firewall (inbound TCP 9997).
• Confirm with your network team that no intermediate devices (e.g., proxies, NATs) are blocking traffic.

Verify Indexer Receiving Configuration

• The indexer must be configured to receive data. Ensure it’s set to listen on the expected port (e.g., 9997).
• Validate Inputs Configuration
• The UF needs to know what data to forward. Check inputs.conf:
• Location: C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf (or an app directory).

Example for Windows Event Logs:

[WinEventLog://Application]
disabled = false
index = my_index

[kiran_panchavat]

@wj742 

• Check Splunk Process: Ensure the Splunk Universal Forwarder (UF) process is running.
• Verify Configuration Files: Ensure inputs.conf and outputs.conf are correctly configured. Here are sample configurations:

# inputs.conf
[monitor://C:\path\to\logs]
disabled = false
sourcetype = my_sourcetype

# outputs.conf
[tcpout:my_indexer]
server = indexer_hostname:9997

• Ensure the Splunk user has the necessary permissions to read the log files being monitored.

[wj742]

1. Splunk process is running on the server.

2. Configured the correct inputs under inputs.conf and outputs. conf

###### OS Logs ######
[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 300
index = wineventlog
renderXml=false

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 300
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist3 = EventCode="5447"
index = wineventlog
renderXml=false

[kiran_panchavat]

@wj742 

Check the splunkd.log for more detailed information.