topic Data is not getting indexed through Universal Forwarder in Getting Data In

Data is not getting indexed through Universal Forwarder

c_krishna_gutur — Tue, 29 Sep 2020 16:32:56 GMT

Hello All, We are forwarding data to indexer from Universal forwarder for couple of months perfectly. Recently we are facing issues that the forwarder is not sending files to indexer and I observed log errors as

10-30-2017 12:29:04.614 +0530 ERROR BTree - 64th child has invalid offset: indexsize=134928 recordsize=291776, (Leaf)
10-30-2017 12:29:04.614 +0530 ERROR BTreeCP - addUpdate CheckValidException caught: BTree::Exception: Validation failed in checkpoint
10-30-2017 12:29:04.676 +0530 ERROR BTree - reading one headers failed: Cannot create a file when that file already exists.
10-30-2017 12:29:04.676 +0530 ERROR BTree - verifyHeaders failed
10-30-2017 12:29:04.676 +0530 ERROR TailReader - Ignoring path="C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage.log" due to: BTree::Exception: failed to restore checkpoint

10-31-2017 14:09:54.581 +0530 ERROR BTreeCP - open failed to restore checkpoint in btree='C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket\splunk_private_db', itmay be corrupted -- run SPLUNK_HOME/bin/btprobe -d '<C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket>' -r to attempt to repair .

Please let me know the actions to remove this error.
Thanks in Advance.

Re: Data is not getting indexed through Universal Forwarder

traxxasbreaker — Thu, 02 Nov 2017 16:52:59 GMT

I'd try running that SPLUNK_HOME/bin/btprobe -d '<C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket>' -r command first and see if it helps. Otherwise, I don't think I've ever encountered a "BTree" error that didn't require reinstalling the forwarder to resolve it, so that is what I'd try next.

Re: Data is not getting indexed through Universal Forwarder

c_krishna_gutur — Fri, 03 Nov 2017 04:22:58 GMT

Thanks for quick reply, I am novice to Splunk and I am worried to run the command that it may end up in any data loss or failure of existing index as client is completely based on these reports. If I run will there be any impact on existing environment, Please suggest.

Re: Data is not getting indexed through Universal Forwarder

traxxasbreaker — Fri, 03 Nov 2017 14:25:38 GMT

On the forwarder, it will cause data still present on the filesystem to be reindexed. The fishbucket is what holds the file checkpoints for what the forwarder is monitoring, so it looks like fishbucket corruption is preventing the forwarder from figuring out where to pick up on the files it is monitoring. If the repair command doesn't work, then a reinstall would have the same effect as it wouldn't preserve the fishbucket, so it would cause all the files to be reread from the beginning.