Unusable Filesystem

[jessieb_83]

I'm setting up a lab instance of  Splunk Ent in prep to replace our legacy instance in a live environment and getting this error message:

"homePath='/mnt/splunk_hot/abc/db' of index=abc on unusable filesystem"

I'm running RHEL 8 VM's, running Splunk 9.1, 2 indexers clustered  together and a cluster manager. I've attached external drives for hot and cold to each indexer.

The external drives have been formatted in ext4 and set in fdisk to mount at boot every time as /mnt/splunk_hot and /mnt/splunk_cold and pointed indexes.conf by volume to them. They come up at boot, I can navigate to them and write to them. They're currently owned by root. I couldn't find who should have permission over them so I left them as is to start.

I tried to enable OPTIMISTIC_ABOUT_FILE_LOCKING=1  but that didn't do anything. That being said, i suspect I've missed a step in the actions taken mounting the external drives. 

I wasn't able to find specifics about the way I'm doing this, so I pose the question: 

Am I doing something wrong, or missing a step on mounting these external drives? Is that now a bad practice? 

I'm stumped.

my indexes.conf:

[volume:hot]
path=/mnt/splunk_hot

[volume:cold]
path=/mnt/splunk_cold

[abc]
repFactor = auto
homePath = volume:hot/abc/db
coldPath = volume:cold/abc/db
thawedPath = $SPLUNK_DB/abc/thaweddb
##We're not utilizing frozen storage at all so I left it default

Any advice here would be greatly appreciated!

[jessieb_83 · Accepted solution]

Finally figured out it was a permission issue. I didn't give splunk ownership over the index locations. 

[gcusello]

Hi @jessieb_83 ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[PickleRick]

My first hint whenever "something strange" happens seemingly at OS level would be of course to check SELinux.

[gcusello]

Hi @jessieb_83,

let me understand: you want to use as $SPLUNK_DB a removable hard drive?

I'm not sure that's possible.

Open a case to Splunk Support, they are the only that can answer to you.

ciao.

Giuseppe

[jessieb_83]

I left the Frozen drive to point to $SPLUNK_DB on the indexer's drive, but I'm not trying to employ frozen buckets at all.

I'm trying to use the volumes on external drives for hot and cold, that's how our current instance is set up. The difference being the current is on Windows, and this new one is going to be on RHEL8.