In order to make the forwarder re-index the entire data. you need to clear the fishbucket. You can do this by deleting $SPLUNK_HOME/var/lib/splunk/fishbucket and restart the forwarder instance. By doing this it will make the forwarder to re-index everything. If you are looking to do this for a single file try adding CrcSalt to your inputs.conf, like crcSalt = readItAgain.
At first check if you Splunk server is receiving logs from your target using a simple search:
index=_internal host=your_host | head 100
checking also last days or always.
If you have results there's an ingestion problem, otherwise a connection problem.
If you haven't results, try with telnet to understand if the connection is open:
telnet ip_server 9997
If ports are open to answer to your question I need of the outputs.conf of your Universal Forwarder (usually is in $SPLUNK_HOME/etc/system/local or in a dedicated App).
If you have results on _internal but not other logs, you should share your inputs.conf (usually is in $SPLUNK_HOME/etc/system/local or in a dedicated App).
I means that I was configured forwarder send data to an incorrect IP address and I was fixed it, now the forwarder could get connection with indexer but not start send data for it was been blocked. So how should I do next to enable data sending on forwarder?
Is it correct?
Some questions or test to perform:
1. I want configure forwarder forward data to 192.168.3.2:9997 but I make a mistike when edit the outputs.conf like follow:
[tcpout:jinmu] server = 192.168.3.2:9998
Then, the following message appears in splunkd.log on forwarder:
10-16-2019 16:56:03.398 +0800 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group jinmu has been blocked for 3900 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
I fixed configuration in outputs.conf:
server = 192.168.3.2:9997
I can't recieve the forwarder data yet. (Maybe for the forwarder is blocked? )
did you checked the connection
telnet 192.168.3.2 9997 ?
did you checked if internal logs arrive to Splunk
index=_internal host=your_host | head 1000 ?
using CLI to restart Splunk, is there any error message?
There is no errors occured when I restart splunk with CLI, and the other 2 forwarders is running well...
At now, I want to know did I need to wait the forwarder block time expired and no the other method to make the block time reset?
no, you don't need the forwarder block time expires.
Telnet is ok?
if you run
index=_internal host=your_host earliest=-7d latest=now | head 1000 have you results?
It missing about 6 hours ago after I restart the forwarder.
In actually, 3 forwarders and indexer are in 4 different LAN, maybe there are some issue occures in the network of missing forwarder.