topic Re: Forwarder Resend Data After Connect To Indexer in Getting Data In

Forwarder Resend Data After Connect To Indexer

aojie654 — Wed, 16 Oct 2019 05:34:17 GMT

Hi, Splunkers:

I have a forwarder that is target to a incorrect indexer and it was paused to send data for 3700s.

Now I have configured to a correct indexer URI and how can I make the forwarder restarting send the data to indexer?

Re: Forwarder Resend Data After Connect To Indexer

gcusello — Wed, 30 Sep 2020 02:33:30 GMT

Hi aojie654,
At first check if you Splunk server is receiving logs from your target using a simple search:

index=_internal host=your_host | head 100

checking also last days or always.
If you have results there's an ingestion problem, otherwise a connection problem.

If you haven't results, try with telnet to understand if the connection is open:

telnet ip_server 9997

If ports are open to answer to your question I need of the outputs.conf of your Universal Forwarder (usually is in $SPLUNK_HOME/etc/system/local or in a dedicated App).

If you have results on _internal but not other logs, you should share your inputs.conf (usually is in $SPLUNK_HOME/etc/system/local or in a dedicated App).

Ciao.
Giuseppe

Re: Forwarder Resend Data After Connect To Indexer

skrajkumar_splu — Wed, 16 Oct 2019 07:19:58 GMT

In order to make the forwarder re-index the entire data. you need to clear the fishbucket. You can do this by deleting $SPLUNK_HOME/var/lib/splunk/fishbucket and restart the forwarder instance. By doing this it will make the forwarder to re-index everything. If you are looking to do this for a single file try adding CrcSalt to your inputs.conf, like crcSalt = readItAgain.

https://docs.splunk.com/Documentation/Splunk/7.3.2/Data/Monitorfilesanddirectorieswithinputs.conf

Re: Forwarder Resend Data After Connect To Indexer

aojie654 — Wed, 16 Oct 2019 07:24:02 GMT

Hi, Giuseppe:

I means that I was configured forwarder send data to an incorrect IP address and I was fixed it, now the forwarder could get connection with indexer but not start send data for it was been blocked. So how should I do next to enable data sending on forwarder?

Re: Forwarder Resend Data After Connect To Indexer

gcusello — Wed, 16 Oct 2019 08:06:46 GMT

Hi aojie654,
let meunderstand:

you configured an incorrect destination IP in your target,
you corrected it,
but your target continues to send data to the wrong Indexer

Is it correct?

Some questions or test to perform:

did you restart Splunk on target after you modified outputs.conf?
did you tried the check I suggested in my eprevious answer?
Then, which outputs.conf did you modify on Target?
The correct Splunk server is correctly configure to receive logs from the target (receiving enabled)?

Ciao.
Giuseppe

Re: Forwarder Resend Data After Connect To Indexer

aojie654 — Wed, 16 Oct 2019 09:08:29 GMT

Emmm...
For example,
1. I want configure forwarder forward data to 192.168.3.2:9997 but I make a mistike when edit the outputs.conf like follow:

[tcpout:jinmu]
server = 192.168.3.2:9998

Then, the following message appears in splunkd.log on forwarder:

10-16-2019 16:56:03.398 +0800 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group jinmu has been blocked for 3900 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
I fixed configuration in outputs.conf:

[tcpout:jinmu]
server = 192.168.3.2:9997
I can't recieve the forwarder data yet. (Maybe for the forwarder is blocked? )

Re: Forwarder Resend Data After Connect To Indexer

aojie654 — Wed, 16 Oct 2019 09:09:15 GMT

I was restarted the forwarder after fixed the configuration file.

Re: Forwarder Resend Data After Connect To Indexer

gcusello — Wed, 16 Oct 2019 09:25:33 GMT

Hi aojie654,
did you checked the connection telnet 192.168.3.2 9997 ?
did you checked if internal logs arrive to Splunk index=_internal host=your_host | head 1000 ?
using CLI to restart Splunk, is there any error message?

Ciao.
Giuseppe

Re: Forwarder Resend Data After Connect To Indexer

aojie654 — Wed, 16 Oct 2019 09:38:32 GMT

There is no errors occured when I restart splunk with CLI, and the other 2 forwarders is running well...

At now, I want to know did I need to wait the forwarder block time expired and no the other method to make the block time reset?

Re: Forwarder Resend Data After Connect To Indexer

gcusello — Wed, 16 Oct 2019 09:49:36 GMT

Hi aojie654,
no, you don't need the forwarder block time expires.
Telnet is ok?
if you run index=_internal host=your_host earliest=-7d latest=now | head 1000 have you results?

Ciao.
Giuseppe

Re: Forwarder Resend Data After Connect To Indexer

aojie654 — Wed, 16 Oct 2019 10:04:03 GMT

Hi, Giuseppe:

It missing about 6 hours ago after I restart the forwarder.

In actually, 3 forwarders and indexer are in 4 different LAN, maybe there are some issue occures in the network of missing forwarder.

Re: Forwarder Resend Data After Connect To Indexer

gcusello — Wed, 16 Oct 2019 10:16:45 GMT

for this reason I asked the two checks
what are the checks results?
ciao.
Giuseppe

Re: Forwarder Resend Data After Connect To Indexer

woodcock — Wed, 16 Oct 2019 21:09:58 GMT

You should not have to do anything. For only 3700 seconds, it should have been able to queue it and then restart where it left off when you added the correct Indexers.