topic Re: Concerns about continuously receiving several error messages in splunkd.log in Monitoring Splunk

Concerns about continuously receiving several error messages in splunkd.log

bkatzlin — Sat, 06 Jun 2020 01:44:28 GMT

Hello,

I'm continuously experiencing several error messages in splunkd.log such as:

ERROR BucketMover - Unable to parse bucket ID from path="..."

In this forum I only found previous solutions saying like "This issue has been fixed in version 6.x" - but I'm currently on 7.2.0

Should I be concerned about these messages?
Would this error affect my data retention?
And finally - how could the issue be fixed?

Regards,
Bernd

Re: Concerns about continuously receiving several error messages in splunkd.log

skalliger — Thu, 27 Jun 2019 18:28:10 GMT

I'd suggest upgrading to the latest 7.2 which is 7.2.7 right now. Don't just do .0 releases.
Do you have other Error log messages around that time when it happens?

Skalli

Re: Concerns about continuously receiving several error messages in splunkd.log

amitm05 — Fri, 28 Jun 2019 08:27:09 GMT

@bkatzlin
You have omitted the bucket path in your question. It'll help to know that which buckets (hot, warm Or cold) are getting impacted.
You might want to check for permissions of Splunk on the storage locations (However if in the first place splunk was able to write there, I'm least expecting to find this to be the cause but no harm cross checking).

And yes, about retention - If your buckets keep failing to roll, you might start experiencing the performance problems and eventually the disk usage.

Ques - Are you working with a idx cluster Or a standalone box ? You might want to check which instances are throwing these errors if you are working with a cluster.

Let me know. Thanks

Re: Concerns about continuously receiving several error messages in splunkd.log

bkatzlin — Fri, 28 Jun 2019 12:36:27 GMT

Thanks for giving me some directions.
Well, it's a idx cluster, Splunk version 7.2.0 as said, and the messages are seen on all cluster members.
Actually the error seems to me related to data model acceleration, since all path="..." names have in common:

.../<index_name>/datamodel_summary/.../DM_Splunk_SA_CIM_Web

Any ideas?

Re: Concerns about continuously receiving several error messages in splunkd.log

bkatzlin — Mon, 01 Jul 2019 14:11:39 GMT

I found that these BucketMover ERROR records are followed by a INFO message saying:
INFO BucketMover - AsyncFreezer freeze succeedded for bkt='...'
So, without knowing any internals, I'd guess that the AsyncFreezer has successfully fixed the affected bucket and everything's ok.

Re: Concerns about continuously receiving several error messages in splunkd.log

skalliger — Mon, 01 Jul 2019 14:36:32 GMT

Guess I forgot to add that. Yea, seems like a bug which might be gone after doing an upgrade. 🙂

Skalli

Re: Concerns about continuously receiving several error messages in splunkd.log

amitm05 — Mon, 01 Jul 2019 14:46:12 GMT

Please provide the full error message. Earlier you didnt mention anything about the DataModel in your error

Re: Concerns about continuously receiving several error messages in splunkd.log

amitm05 — Wed, 03 Jul 2019 06:12:59 GMT

I'd suggest to mark the answer if this problem is getting auto fixed. So its easier for others to follow up on the correct thread. Thanks

Re: Concerns about continuously receiving several error messages in splunkd.log

bkatzlin — Wed, 03 Jul 2019 09:52:12 GMT

Thanks to Skalli's comment I was able to find a INFO message corresponding to each ERROR, saying that the bucket issue has somehow been resolved by "AsyncFreezer". I should have noticed this immediately, sorry.
Thus I'd prefer to take these errors easy and see whether they will disappear at the next upgrade.
Thanks!

Re: Concerns about continuously receiving several error messages in splunkd.log

skalliger — Wed, 03 Jul 2019 11:40:42 GMT

Good to know, thanks for the feedback. Sometimes there are errors that shouldn't be displayed as errors. 🙂

Skalli