Most of the time, we are seeing that the Splunk universal forwarder or heavy forwarder is failing to forward data to the indexer. In this scenario, what troubleshooting steps should we take to investigate why this is happening?
check if Splunk process is running on Splunk forwarder
For Windows check services | for Linux use below command
ps -ef |grep splunkd
cd $SPLUNK HOME/bin
Check if Splunk forwarder forwarding port is open by using below command
netstat -an | grep 9997
If output of above command is blank, then your port is not open. You need to open it.
Check on indexer if receiving is enabled on port 9997 and port 9997 is open on indexer
Check if receiving is configured : on indexer, go to setting>>forwarding and receiving >> check if receiving is enabled on port 9997. If not, enable it.
Check if you are able to ping indexer from forwarder host
ping indexer name
If you are not able to ping to the server, then check network issue
Confirm on indexer if your file is already indexed or not by using the below search query
In the Splunk UI, run the following search - index=_internal "FileInputTracker" **
As output of the search query, you will get a list of log files indexed.
Check if forwarder has completed processing log file (i.e. tailing process by using below URL)
https://splunk forwarder server name:8089/services/admin/inputstatus/TailingProcessor:FileStatus
In tailing process output you can check if forwarder is having an issue for processing file
Check out log file permissions which you are sending to Splunk. Verify if Splunk user has access to log file
Checkout filesystem for last modification and verify if the forwarder is monitoring it
Verify inputs.conf and outputs.conf for proper configuration
Below are sample configuration files for comparison:
disabled = false
sourcetype = linux_secure
disabled = false
server = impAserver01.domain:9997,impAserver02.domain:9997
autoLB = true
defaultGroup = impB
server = impBserver01.domain:9997,impBserver02.domain:9997
Checkout disk space availability on the indexer
**Check splunkd.log on forwarder at location $SPLUNK_HOME/var/log/splunk for any errors. Like for messages that are from 'TcpOutputProc', they should give you an indication as to what is occurring when the forwarder tries to connect to the indexer
tcpdump port 9997 data for any errors
tcpdump -i etho port 9997
Check out ulimit if you have installed forwarder on linux. and set it to unlimites or max (65535 -Splunk recommended)
- ulimit is limit set by default in linux is limit for number files opened by a process
- check ulimit command: ulimit -n
- set ulimit command: ulimit -n expected size
Finally, try restarting Splunk on the forwarder
@SloshBurch This could maybe be
Yup! Actually our friends in Professional Services have a similar article that we've been talking about publishing.
Thanks for drawing my attention!
Hi dkolekar [Splunk],
to perform troubleshooting on Splunk data ingestion you can see at
Anyway, if you're not receiving any log from UF you could perform these tests:
If instead the problem is that UF is sending data (there are events in _internal) but there arent logs, check the log path and file names, probably they are wrong.
Check also the timestamp format, maybe UF failed in timestamp parsing and you have your logs with a wrong timestamp.
In other words, fix your checks to be sure that:
The above documentation can support you in your debugging.