topic Re: Splunk Smartstore: Remote Storage Http Error Status Codes in Deployment Architecture

Splunk Smartstore: Remote Storage Http Error Status Codes

saiganesh49 — Wed, 21 Aug 2019 16:48:14 GMT

I have configured Smart Store on our Splunk cluster recently, after the changes,
I was seeing 403 and 404 errors in Remote Storage Http Error Status Codes.

Is anyone had the similar issue ?

Re: Splunk Smartstore: Remote Storage Http Error Status Codes

jbuckner85 — Mon, 16 Sep 2019 16:19:24 GMT

Did you ever find a resolution to this issue? We are encountering the same thing.

Re: Splunk Smartstore: Remote Storage Http Error Status Codes

rwsisson — Mon, 16 Sep 2019 16:25:31 GMT

No, we did not

Re: Splunk Smartstore: Remote Storage Http Error Status Codes

saiganesh49 — Mon, 16 Sep 2019 16:56:35 GMT

posted in answer

Re: Splunk Smartstore: Remote Storage Http Error Status Codes

cudgel — Thu, 19 Sep 2019 15:01:38 GMT

We are seeing the same thing in our environment. 403's doing REST.GET.BUCKETVERSIONS and 404's on REST.HEAD.OBJECT for keys ending in "receipts.json" or "done". Searches don't seem to be having problems, but we have a very high error count.

Re: Splunk Smartstore: Remote Storage Http Error Status Codes

saiganesh49 — Thu, 19 Sep 2019 15:03:08 GMT

which version of Splunk are you on ?

Re: Splunk Smartstore: Remote Storage Http Error Status Codes

cudgel — Thu, 19 Sep 2019 15:22:14 GMT

version 7.2.5.1

Re: Splunk Smartstore: Remote Storage Http Error Status Codes

rwsisson — Thu, 19 Sep 2019 16:24:01 GMT

version 7.3.0

Re: Splunk Smartstore: Remote Storage Http Error Status Codes

srajarat2 — Wed, 30 Sep 2020 02:51:26 GMT

Based on our troubleshooting, Splunk is issuing a get_object of receipt.json (with the whole bucket details in the object name) before even performing a put_object and this should result in 404. As per Splunk support this can be ignored unless you see 404 for the same object multiple times.

Re: Splunk Smartstore: Remote Storage Http Error Status Codes

saiganesh49 — Fri, 08 Nov 2019 01:32:51 GMT

{
    "Version": "***",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetLifecycleConfiguration",
                "s3:DeleteObjectVersion",
                "s3:ListBucketVersions",
                "s3:GetBucketLogging",
                "s3:RestoreObject",
                "s3:ListBucket",
                "s3:GetBucketVersioning",
                "s3:PutObject",
                "s3:GetObject",
                "s3:PutLifecycleConfiguration",
                "s3:GetBucketCORS",
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::your-smartstore-bucket-name/*",
                "arn:aws:s3:::your-smartstore-bucket-name"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}

this bucket policy has zeroed my 403 errors , as far as 404 errors Splunk confirmed that this will be fixed in the future releases.