topic Re: How do I resolve this error in index: _internal? in Splunk Search

How do I resolve this error in index: _internal?

Taruchit — Mon, 24 Apr 2023 19:03:45 GMT

Hello All,

I am searching for corrupt data in Splunk, and thus executed the below query: -

index=_internal sourcetype=splunk_search_messages "corrupt" OR "corrupted"

I got the below errors: -

Error 1: -

message=[wd*****] [subsearch]: Failed to read size=4 event(s) from rawdata in bucket='dummyIndex~119~8X88XXY-X8XX-88X8-888X-X88X88XX8888' path='/u01/ovz/data/dummyIndex/db/db_1611661166_1611222333_111_8X88XX8X-X8XX-88X8-888X-X88X88XX8888. Rawdata may be corrupt, see search.log. Results may be incomplete!

Error 2: -

01-01-2023 07:01:01.098 ERROR SRSSerializer [12729 RemoteTimelineReadThread] - cannot read file magic -probably corrupt

Error 3: -

Error decompressing zstd block: Corrupted block detected

I need your help to understand and get details about the above errors, Error 1, Error 2 and Error 3.

In Error 1, it states to check search.log. Thus, it would be helpful if you can share how to fetch the relevant information from the file.

So far for error 1, I found https://community.splunk.com/t5/Splunk-Search/Searchquery-error/m-p/509508 which states that file may be corrupt.

Any information about the above two errors will be very helpful.

Thank you

Re: How do I resolve this Error in index: _internal?

Taruchit — Mon, 24 Apr 2023 18:39:30 GMT

For error 1, I fetched following details on https://docs.splunk.com/Documentation/Splunk/9.0.4/Indexer/HowSplunkstoresindexes: -

db_1611661166_1611222333_111_8X88XX8X-X8XX-88X8-888X-X88X88XX8888

db: Originating bucket
1611661166: Newest time
1611222333: Oldest time
111: localid; ID for the bucket
8X88XX8X-X8XX-88X8-888X-X88X88XX8888: guid; guid of the source peer node

Re: How do I resolve this error in index: _internal?

richgalloway — Mon, 24 Apr 2023 20:05:56 GMT

Error #1 should not reference the search log since there is nothing in the message that tells us what search is reported the error. You can ignore that suggestion. You can, however, use the fsck command to attempt to repair the file.

The other two error messages do not provide enough information to diagnose or fix anything. Unless there are other messages that offer more context or information then you can ignore the error.

Re: How do I resolve this Error in index: _internal?

richgalloway — Mon, 24 Apr 2023 20:07:02 GMT

That is the correct breakdown of a bucket name.