topic Experiencing [SmartStore] HTTP 502 network errors with S3? in Knowledge Management

Experiencing [SmartStore] HTTP 502 network errors with S3?

serge_ohpen — Fri, 18 Mar 2022 14:47:11 GMT

Hello All,

After configuring migration for a few indexes, the following errors is filling up the log on all cluster peers:

06-27-2019 07:43:11.914 +0200 ERROR S3Client - command=get transactionId=0x4053369110 rTxnId=0x4051be42d0 status=comple ted success=N localPath="D:\Program Files\Splunk\var\lib\splunk\db\db_1561365505_1561358169_598_A3573547-8422-44 29-B22D-4867A0A1B8E8.tmp" offset=0 error="can not open file" reason="Access is denied." 06-27-2019 07:43:11.915 +0200 ERROR RetryableClientTransaction - transactionDone(): groupId=0x4043b50ef0 rTxnId=0x4051b e42d0 transactionId=0x4053369110 success=N HTTP-statusCode=502 HTTP-statusDescription="network error" retries=429496729 5 retry=N no_retry_reason="transaction group had fatal error" remainingTxns=0

S3 bucket is populated with files and retrieving also works - it's just amount of errors is insane

We're running EC2 instances, is there a way we can find why are we getting Bad Gateway errors in Splunk and how to solve them?

Thx, Serge.

Re: [SmartStore] HTTP 502 network errors with S3?

skalliger — Thu, 27 Jun 2019 16:41:21 GMT

Always try to keep the Windows path as short as possible, e.g. install to E:\splunk.

There is a permission error in your event provided. Does your splunk user have admin permissions? That's usually needed for the Program Files folder.

Skalli

Re: [SmartStore] HTTP 502 network errors with S3?

serge_ohpen — Fri, 28 Jun 2019 04:58:26 GMT

The permissions are correct but just in case I have reset them yet still getting access denied messages. What is strange, I can see .tmp files been created there with 0 size and then disappear again.

Thx, Serge.

Re: [SmartStore] HTTP 502 network errors with S3?

rbal_splunk — Fri, 30 Aug 2019 22:33:12 GMT

With Splunk smartstore Generally, ignore HTTP status code on their own

There could be intermittent problems in these codes which get resolved by a retry.
The codes become valuable when investigating other issues.
They could raise red flags if some HTTP status codes start appearing, such as 401's, for example. It wouldn't be practical to enumerate which codes and with which frequency indicates a problem.

Re: [SmartStore] HTTP 502 network errors with S3?

srajarat2 — Wed, 06 Nov 2019 19:46:41 GMT

Seems like Splunk SmartStore is not supported on Windows, yet?

Check this link.
https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/SmartStoresystemrequirements#Operating_system_requirements

It refers to Linux only.

Re: [SmartStore] HTTP 502 network errors with S3?

srajarat2 — Fri, 08 Nov 2019 00:31:52 GMT

Splunk Support did confirm that SmartStore is only supported on Linux at this time and not on Windows.

Re: [SmartStore] HTTP 502 network errors with S3?

thormanrd — Fri, 18 Mar 2022 13:49:22 GMT

There is fundamentally a problem ignoring HTTP responses. Many issues are illuminated with these, such as:

HTTP 503 Slow down - typically a keyspace design issue
HTTP 502 Read timeout - long reads and head operations with too little timeout setting
HTTP 502 Connection timeout - long or busy network connections with too little timeout setting
HTTP 500 Internal server error - poor exception processing that needs bug report
HTTP 502 Name or Service not known - failure in DNS services
HTTP 502 Partial upload - failure in multi-part upload operation that needs bug report