Experiencing [SmartStore] HTTP 502 network errors with S3?

[serge_ohpen]

Hello All,

After configuring migration for a few indexes, the following errors is filling up the log on all cluster peers:

06-27-2019 07:43:11.914 +0200 ERROR S3Client - command=get transactionId=0x4053369110 rTxnId=0x4051be42d0 status=comple
ted success=N localPath="D:\Program Files\Splunk\var\lib\splunk\db\db_1561365505_1561358169_598_A3573547-8422-44
29-B22D-4867A0A1B8E8.tmp" offset=0 error="can not open file" reason="Access is denied."
06-27-2019 07:43:11.915 +0200 ERROR RetryableClientTransaction - transactionDone(): groupId=0x4043b50ef0 rTxnId=0x4051b
e42d0 transactionId=0x4053369110 success=N HTTP-statusCode=502 HTTP-statusDescription="network error" retries=429496729
5 retry=N no_retry_reason="transaction group had fatal error" remainingTxns=0

S3 bucket is populated with files and retrieving also works - it's just amount of errors is insane

We're running EC2 instances, is there a way we can find why are we getting Bad Gateway errors in Splunk and how to solve them?

Thx, Serge.

[srajarat2]

Splunk Support did confirm that SmartStore is only supported on Linux at this time and not on Windows.

[srajarat2]

Seems like Splunk SmartStore is not supported on Windows, yet?

Check this link.
https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/SmartStoresystemrequirements#Operating_sy...

It refers to Linux only.

[rbal_splunk]

With Splunk smartstore Generally, ignore HTTP status code on their own

• There could be intermittent problems in these codes which get resolved by a retry.
• The codes become valuable when investigating other issues.
• They could raise red flags if some HTTP status codes start appearing, such as 401's, for example. It wouldn't be practical to enumerate which codes and with which frequency indicates a problem.
  

[thormanrd]

There is fundamentally a problem ignoring HTTP responses.  Many issues are illuminated with these, such as:

1. HTTP 503 Slow down - typically a keyspace design issue
2. HTTP 502 Read timeout - long reads and head operations with too little timeout setting
3. HTTP 502 Connection timeout - long or busy network connections with too little timeout setting
4. HTTP 500 Internal server error - poor exception processing that needs bug report
5. HTTP 502 Name or Service not known - failure in DNS services
6. HTTP 502 Partial upload - failure in multi-part upload operation that needs bug report

[skalliger]

Always try to keep the Windows path as short as possible, e.g. install to E:\splunk.

There is a permission error in your event provided. Does your splunk user have admin permissions? That's usually needed for the Program Files folder.

Skalli

[serge_ohpen]

The permissions are correct but just in case I have reset them yet still getting access denied messages. What is strange, I can see .tmp files been created there with 0 size and then disappear again.

Thx, Serge.