Getting Data In

Forwarder Resend Data After Connect To Indexer

aojie654
Path Finder

Hi, Splunkers:

I have a forwarder that is target to a incorrect indexer and it was paused to send data for 3700s.

Now I have configured to a correct indexer URI and how can I make the forwarder restarting send the data to indexer?

0 Karma

woodcock
Esteemed Legend

You should not have to do anything. For only 3700 seconds, it should have been able to queue it and then restart where it left off when you added the correct Indexers.

0 Karma

skrajkumar_splu
Splunk Employee
Splunk Employee

In order to make the forwarder re-index the entire data. you need to clear the fishbucket. You can do this by deleting $SPLUNK_HOME/var/lib/splunk/fishbucket and restart the forwarder instance. By doing this it will make the forwarder to re-index everything. If you are looking to do this for a single file try adding CrcSalt to your inputs.conf, like crcSalt = readItAgain.

https://docs.splunk.com/Documentation/Splunk/7.3.2/Data/Monitorfilesanddirectorieswithinputs.conf

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aojie654,
At first check if you Splunk server is receiving logs from your target using a simple search:

index=_internal host=your_host | head 100

checking also last days or always.
If you have results there's an ingestion problem, otherwise a connection problem.

If you haven't results, try with telnet to understand if the connection is open:

telnet ip_server 9997

If ports are open to answer to your question I need of the outputs.conf of your Universal Forwarder (usually is in $SPLUNK_HOME/etc/system/local or in a dedicated App).

If you have results on _internal but not other logs, you should share your inputs.conf (usually is in $SPLUNK_HOME/etc/system/local or in a dedicated App).

Ciao.
Giuseppe

0 Karma

aojie654
Path Finder

Hi, Giuseppe:

I means that I was configured forwarder send data to an incorrect IP address and I was fixed it, now the forwarder could get connection with indexer but not start send data for it was been blocked. So how should I do next to enable data sending on forwarder?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aojie654,
let meunderstand:

  • you configured an incorrect destination IP in your target,
  • you corrected it,
  • but your target continues to send data to the wrong Indexer

Is it correct?

Some questions or test to perform:

  • did you restart Splunk on target after you modified outputs.conf?
  • did you tried the check I suggested in my eprevious answer?
  • Then, which outputs.conf did you modify on Target?
  • The correct Splunk server is correctly configure to receive logs from the target (receiving enabled)?

Ciao.
Giuseppe

0 Karma

aojie654
Path Finder

Emmm...
For example,
1. I want configure forwarder forward data to 192.168.3.2:9997 but I make a mistike when edit the outputs.conf like follow:

[tcpout:jinmu]
server = 192.168.3.2:9998
  1. Then, the following message appears in splunkd.log on forwarder:

    10-16-2019 16:56:03.398 +0800 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group jinmu has been blocked for 3900 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

  2. I fixed configuration in outputs.conf:

    [tcpout:jinmu]
    server = 192.168.3.2:9997

  3. I can't recieve the forwarder data yet. (Maybe for the forwarder is blocked? )

0 Karma

aojie654
Path Finder

I was restarted the forwarder after fixed the configuration file.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aojie654,
did you checked the connection telnet 192.168.3.2 9997 ?
did you checked if internal logs arrive to Splunk index=_internal host=your_host | head 1000 ?
using CLI to restart Splunk, is there any error message?

Ciao.
Giuseppe

0 Karma

aojie654
Path Finder

There is no errors occured when I restart splunk with CLI, and the other 2 forwarders is running well...

At now, I want to know did I need to wait the forwarder block time expired and no the other method to make the block time reset?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aojie654,
no, you don't need the forwarder block time expires.
Telnet is ok?
if you run index=_internal host=your_host earliest=-7d latest=now | head 1000 have you results?

Ciao.
Giuseppe

0 Karma

aojie654
Path Finder

Hi, Giuseppe:

It missing about 6 hours ago after I restart the forwarder.

In actually, 3 forwarders and indexer are in 4 different LAN, maybe there are some issue occures in the network of missing forwarder.

0 Karma

gcusello
SplunkTrust
SplunkTrust

for this reason I asked the two checks
what are the checks results?
ciao.
Giuseppe

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.