How do you get a universal forwarder to send missing data?

[michael_leo]

We had a weird incident happen and we stopped receiving log files for a very specific time window.

Is there a way to kick the UF clients to resend data to the indexers?

[renjith_nair]

@michael_leo ,

You can resend old data by cleaning fishbucket index where Splunk software tracks how far into a file indexing has progressed, to enable the software to detect when data has been added to the file and resume indexing

Either you can remove the files under $SPLUNK_HOME/var/lib/splunk/fishbucket for re-indexing all the inputs from that forwarder or you can use btprobe to selectively re-index specific files.

You need to stop the forwarder while doing the activities.

Have a look at this post for more information - how-to-reindex-data-from-a-forwarder
If you are re-indexing all the inputs, there could create duplicate events in your index and also this might count against your license.

[mykol_j]

FYI,

the clearing fishbucket trick does not seem to work on UFs... at least on 8.2

:/opt/splunkforwarder/etc/system/local# /opt/splunkforwarder/bin/splunk clean eventdata -index _thefishbucket
This action will permanently erase all events from the index '_thefishbucket'; it cannot be undone.
Are you sure you want to continue [y/n]? y
ERROR: Cleaning eventdata is not supported on this version.

[isoutamo]

Hi

another options are:

• remove individual files by: 

  splunk btprobe -d SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/access.log --reset

  You should do this for all individual files which you want to reindexing
• or just remove the fishbucket directory from UF if you want reindex all files

In both cases you must first stop UF before try above commands.

r. Ismo

[mykol_j]

Thanks!

I also discovered that the brute force technique works too:

rm -rf ../var/lib/splunk/fishbucket

Not as elegante, but effective...