Splunk Smartstore: Remote Storage Http Error Status Codes

[saiganesh49]

I have configured Smart Store on our Splunk cluster recently, after the changes,
I was seeing 403 and 404 errors in Remote Storage Http Error Status Codes.

Is anyone had the similar issue ?

[saiganesh49 · Accepted solution]

{
    "Version": "***",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetLifecycleConfiguration",
                "s3:DeleteObjectVersion",
                "s3:ListBucketVersions",
                "s3:GetBucketLogging",
                "s3:RestoreObject",
                "s3:ListBucket",
                "s3:GetBucketVersioning",
                "s3:PutObject",
                "s3:GetObject",
                "s3:PutLifecycleConfiguration",
                "s3:GetBucketCORS",
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::your-smartstore-bucket-name/*",
                "arn:aws:s3:::your-smartstore-bucket-name"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}

this bucket policy has zeroed my 403 errors , as far as 404 errors Splunk confirmed that this will be fixed in the future releases.

[cudgel]

We are seeing the same thing in our environment. 403's doing REST.GET.BUCKETVERSIONS and 404's on REST.HEAD.OBJECT for keys ending in "receipts.json" or "done". Searches don't seem to be having problems, but we have a very high error count.

[saiganesh49]

which version of Splunk are you on ?

[cudgel]

version 7.2.5.1

[rwsisson]

version 7.3.0

[srajarat2]

Based on our troubleshooting, Splunk is issuing a get_object of receipt.json (with the whole bucket details in the object name) before even performing a put_object and this should result in 404. As per Splunk support this can be ignored unless you see 404 for the same object multiple times.

[jbuckner85]

Did you ever find a resolution to this issue? We are encountering the same thing.

[rwsisson]

No, we did not

[saiganesh49]

posted in answer