topic Splunk Stuck Checking Indexes when using Smart Store in Splunk Enterprise

Splunk Stuck Checking Indexes when using Smart Store

luke10 — Wed, 20 Mar 2024 22:55:01 GMT

I have a Splunk instance that is deployed on EBS Volume mounted to EC2 Instance.

I started working on enabling Smart Store for one of my indexes but whenever I have the indexes.conf configured to let one of my indexes use the smart store, when I restart splunk it basically hangs on this step:

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...

Nothing found in logs, I am just puzzled how to fix this. Can anybody hint what can be the issue?

indexes.conf:

[volume:s3volumeone]
storageType = remote
path = s3://some-bucket-name
remote.s3.endpoint = https://s3.us-west-2.amazonaws.com

[smart_store_index_10]
remotePath = volume:s3volumeone/$_index_name
homePath = $SPLUNK_DB/$_index_name/db
coldPath = $SPLUNK_DB/$_index_name/colddb
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
maxGlobalDataSizeMB = 0 
maxGlobalRawDataSizeMB = 0 
homePath.maxDataSizeMB = 1000
maxHotBuckets = 2
maxDataSize = 3
maxWarmDBCount = 5
frozenTimePeriodInSecs = 10800

small numbers for bucket size etc. are intentional to allow quick testing of settings.

Re: Splunk Stuck Checking Indexes when using Smart Store

isoutamo — Fri, 22 Mar 2024 18:30:46 GMT

Have you check that your IAM roles are still correctly defined and valid for your node?
Can you run those S3 access test commands?

Re: Splunk Stuck Checking Indexes when using Smart Store

luke10 — Tue, 26 Mar 2024 16:30:30 GMT

I am sorry but I don't see any commands. Did you mean to attach them to the post?

Re: Splunk Stuck Checking Indexes when using Smart Store

isoutamo — Tue, 26 Mar 2024 19:08:12 GMT

I didn’t add those to previous messages as answered by phone.
You could look those from https://docs.splunk.com/Documentation/Splunk/9.2.0/Indexer/TroubleshootSmartStore