Encountering a "command not found" error when executing rebuild command

[adnankhan5133]

Hello,

I executed the below command on an indexer but received a "rebuild: command not found" error message:

splunk rebuild "app/splunk/var/lib/splunk/indexname/thaweddb/

The file name was the name of the frozen bucket that was copied over the Thawed directory since I'm trying to restore archived data from one bucket.

How can I execute this command without error? It appears that Splunk isn't recognizing the rebuild command for some reason.

[MuS · Accepted solution]

This is a different error this time 😉 Try it like this:

/full/path/to/splunk/bin/splunk rebuild app/splunk/var/lib/splunk/cisco_ios/thaweddb/rb_1513987154_1513901000_D4AJ321

Also, using the bucket path without leading / will assume the app directory is in the directory you are executing the command. Use full paths to be sure.

cheers, MuS

[adnankhan5133]

Thanks, that command actually worked!

While this is probably a separate question, I seem to be running into parsing issues with the bucket after successfully executing the splunk rebuild command:

Unable to parse bucket name for bucketType=app/splunk/var/lib/splunk/cisco_ios/thaweddb/rb_1513987154_1513901000_D4AJ321

Is this a bug of some sort? The bucket should be searchable now, but it's not.

[MuS]

Nope, no bug.

Splunk tells you it cannot parse the name of the bucket. It might be as easy as renaming app/splunk/var/lib/splunk/cisco_ios/thaweddb/rb_1513987154_1513901000_D4AJ321 to app/splunk/var/lib/splunk/cisco_ios/thaweddb/db_1513987154_1513901000_D4AJ321

But reading this just now, this name looks weird... bucket names are usually [r|d]b_latest epoch_earliest epoch_bucketid_Splunk instance GUID since you are using a bucket called rb_* it indicates it is from a index cluster and therefore the bucket name is incorrect.

cheers, MuS

[adnankhan5133]

Hmm, so I tried again with a different log source (Linux) and took the following steps. I had a copy/paste error when I previously pasted the bucket name.

Steps:

1) cp -r db_1497345908_1495723992_122_5CA73C89-0181-4ADB-B618-341094395DA1 /app/splunk/var/lib/splunk/linux/thaweddb
--- need to go to the frozen bucket of linux (/backup/splunk/frozen/linux/frozen)
2) /app/splunk/bin/splunk rebuild /app/splunk/var/lib/splunk/linux/thaweddb/db_1497345908_1495723992_122_5CA73C89-0181-4ADB-B618-341094395DA1

--- Rebuild was successful and Splunk indexer was restarted

However, I encountered the same error when checking in the search console:

06-19-2018 00:19:44.521 -0700 ERROR BucketReplicator - Unable to parse bucket name for bucketType=/app/splunk/var/lib/splunk/linux/thaweddb/db_1497345908_1495723992_122_5CA73C89-0181-4ADB-B618-341094395DA1

The naming convention for the archived buckets is actually the same across the board. The archived bucket was retrieved from a filepath called /backup/splunk/frozen/linux/frozen. The bucket would contain data that is around 5 months old - this is what I'm attempting to restore in our search head which only stores the last 3 months of data.

Any help/direction would be greatly appreciated!

[MuS]

Okay, now that you provided the full error message it looks like you hit a know feature - err bug SPL-90468:Clustering: can't replicate thawed buckets - https://answers.splunk.com/answers/153341/thawed-buckets-error-clusterslavebuckethandler-failed-to-t...

But this is not related to your search, can you put it into a dev/test instance, and into a brand new empty index to see if this shows anything over all time?

[MuS]

Might just be a copy / paste error, but you are missing the closing double quotes. Anyway double quotes should not be necessary here, use the full path and the bucket directory instead :

 splunk rebuild /full/path/to/splunk/var/lib/splunk/indexname/thaweddb/bucketnamehere

cheers, MuS

[adnankhan5133]

Hi MuS,

I tried again without the quotes since those aren't needed:

splunk rebuild app/splunk/var/lib/splunk/cisco_ios/thaweddb/rb_1513987154_1513901000_D4AJ321

That's the full path to the frozen bucket which has been copied to the thawed directory and I have been following the instructions written here under the section "Thaw a 4.2+ archive": https://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Restorearchiveddata

However, I got the below error

bash: splunk: command not found