Getting Data In

Data is not getting indexed through Universal Forwarder

c_krishna_gutur
Explorer

Hello All, We are forwarding data to indexer from Universal forwarder for couple of months perfectly. Recently we are facing issues that the forwarder is not sending files to indexer and I observed log errors as

10-30-2017 12:29:04.614 +0530 ERROR BTree - 64th child has invalid offset: indexsize=134928 recordsize=291776, (Leaf)
10-30-2017 12:29:04.614 +0530 ERROR BTreeCP - addUpdate CheckValidException caught: BTree::Exception: Validation failed in checkpoint
10-30-2017 12:29:04.676 +0530 ERROR BTree - reading one headers failed: Cannot create a file when that file already exists.
10-30-2017 12:29:04.676 +0530 ERROR BTree - verifyHeaders failed
10-30-2017 12:29:04.676 +0530 ERROR TailReader - Ignoring path="C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage.log" due to: BTree::Exception: failed to restore checkpoint

10-31-2017 14:09:54.581 +0530 ERROR BTreeCP - open failed to restore checkpoint in btree='C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket\splunk_private_db', itmay be corrupted -- run SPLUNK_HOME/bin/btprobe -d '<C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket>' -r to attempt to repair .

Please let me know the actions to remove this error.
Thanks in Advance.

0 Karma

traxxasbreaker
Communicator

I'd try running that SPLUNK_HOME/bin/btprobe -d '<C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket>' -r command first and see if it helps. Otherwise, I don't think I've ever encountered a "BTree" error that didn't require reinstalling the forwarder to resolve it, so that is what I'd try next.

0 Karma

c_krishna_gutur
Explorer

Thanks for quick reply, I am novice to Splunk and I am worried to run the command that it may end up in any data loss or failure of existing index as client is completely based on these reports. If I run will there be any impact on existing environment, Please suggest.

0 Karma

traxxasbreaker
Communicator

On the forwarder, it will cause data still present on the filesystem to be reindexed. The fishbucket is what holds the file checkpoints for what the forwarder is monitoring, so it looks like fishbucket corruption is preventing the forwarder from figuring out where to pick up on the files it is monitoring. If the repair command doesn't work, then a reinstall would have the same effect as it wouldn't preserve the fishbucket, so it would cause all the files to be reread from the beginning.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Build the Future of Agentic AI: Join the Splunk Agentic Ops Hackathon

AI is changing how teams investigate incidents, detect threats, automate workflows, and build intelligent ...

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...