Knowledge Management
Highlighted

[SmartStore] HTTP 502 network errors with S3?

New Member

Hello All,

After configuring migration for a few indexes, the following errors is filling up the log on all cluster peers:

06-27-2019 07:43:11.914 +0200 ERROR S3Client - command=get transactionId=0x4053369110 rTxnId=0x4051be42d0 status=comple
ted success=N localPath="D:\Program Files\Splunk\var\lib\splunk\db\db15613655051561358169598A3573547-8422-44
29-B22D-4867A0A1B8E8.tmp" offset=0 error="can not open file" reason="Access is denied."
06-27-2019 07:43:11.915 +0200 ERROR RetryableClientTransaction - transactionDone(): groupId=0x4043b50ef0 rTxnId=0x4051b
e42d0 transactionId=0x4053369110 success=N HTTP-statusCode=502 HTTP-statusDescription="network error" retries=429496729
5 retry=N noretryreason="transaction group had fatal error" remainingTxns=0

S3 bucket is populated with files and retrieving also works - it's just amount of errors is insane

We're running EC2 instances, is there a way we can find why are we getting Bad Gateway errors in Splunk and how to solve them?

Thx, Serge.

0 Karma
Highlighted

Re: [SmartStore] HTTP 502 network errors with S3?

SplunkTrust
SplunkTrust

Always try to keep the Windows path as short as possible, e.g. install to E:\splunk.

There is a permission error in your event provided. Does your splunk user have admin permissions? That's usually needed for the Program Files folder.

Skalli

0 Karma
Highlighted

Re: [SmartStore] HTTP 502 network errors with S3?

New Member

The permissions are correct but just in case I have reset them yet still getting access denied messages. What is strange, I can see .tmp files been created there with 0 size and then disappear again.

Thx, Serge.

0 Karma
Highlighted

Re: [SmartStore] HTTP 502 network errors with S3?

Splunk Employee
Splunk Employee

With Splunk smartstore Generally, ignore HTTP status code on their own

  • There could be intermittent problems in these codes which get resolved by a retry.
  • The codes become valuable when investigating other issues.
  • They could raise red flags if some HTTP status codes start appearing, such as 401's, for example. It wouldn't be practical to enumerate which codes and with which frequency indicates a problem.
0 Karma
Highlighted

Re: [SmartStore] HTTP 502 network errors with S3?

Explorer

Seems like Splunk SmartStore is not supported on Windows, yet?

Check this link.
https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/SmartStoresystemrequirements#Operating_sy...

It refers to Linux only.

0 Karma
Highlighted

Re: [SmartStore] HTTP 502 network errors with S3?

Explorer

Splunk Support did confirm that SmartStore is only supported on Linux at this time and not on Windows.

0 Karma