topic Re: How do you get a universal forwarder to send missing data? in Getting Data In

How do you get a universal forwarder to send missing data?

michael_leo — Wed, 13 Feb 2019 21:22:06 GMT

We had a weird incident happen and we stopped receiving log files for a very specific time window.

Is there a way to kick the UF clients to resend data to the indexers?

Re: How do you get a universal forwarder to send missing data?

renjith_nair — Thu, 14 Feb 2019 03:58:42 GMT

@michael_leo ,

You can resend old data by cleaning fishbucket index where Splunk software tracks how far into a file indexing has progressed, to enable the software to detect when data has been added to the file and resume indexing

Either you can remove the files under $SPLUNK_HOME/var/lib/splunk/fishbucket for re-indexing all the inputs from that forwarder or you can use btprobe to selectively re-index specific files.

You need to stop the forwarder while doing the activities.

Have a look at this post for more information - how-to-reindex-data-from-a-forwarder
If you are re-indexing all the inputs, there could create duplicate events in your index and also this might count against your license.

Re: How do you get a universal forwarder to send missing data?

mykol_j — Tue, 07 Jun 2022 16:25:38 GMT

FYI,

the clearing fishbucket trick does not seem to work on UFs... at least on 8.2

:/opt/splunkforwarder/etc/system/local# /opt/splunkforwarder/bin/splunk clean eventdata -index _thefishbucket
This action will permanently erase all events from the index '_thefishbucket'; it cannot be undone.
Are you sure you want to continue [y/n]? y
ERROR: Cleaning eventdata is not supported on this version.

Re: How do you get a universal forwarder to send missing data?

isoutamo — Wed, 08 Jun 2022 08:04:54 GMT

another options are:

remove individual files by:

splunk btprobe -d SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/access.log --reset

You should do this for all individual files which you want to reindexing

or just remove the fishbucket directory from UF if you want reindex all files

In both cases you must first stop UF before try above commands.

r. Ismo

Re: How do you get a universal forwarder to send missing data?

mykol_j — Wed, 08 Jun 2022 14:52:09 GMT

Thanks!

I also discovered that the brute force technique works too:

rm -rf ../var/lib/splunk/fishbucket

Not as elegante, but effective...