Knowledge Management

Experiencing [SmartStore] HTTP 502 network errors with S3?

serge_ohpen
New Member

Hello All,

After configuring migration for a few indexes, the following errors is filling up the log on all cluster peers:

06-27-2019 07:43:11.914 +0200 ERROR S3Client - command=get transactionId=0x4053369110 rTxnId=0x4051be42d0 status=comple
ted success=N localPath="D:\Program Files\Splunk\var\lib\splunk\db\db_1561365505_1561358169_598_A3573547-8422-44
29-B22D-4867A0A1B8E8.tmp" offset=0 error="can not open file" reason="Access is denied."
06-27-2019 07:43:11.915 +0200 ERROR RetryableClientTransaction - transactionDone(): groupId=0x4043b50ef0 rTxnId=0x4051b
e42d0 transactionId=0x4053369110 success=N HTTP-statusCode=502 HTTP-statusDescription="network error" retries=429496729
5 retry=N no_retry_reason="transaction group had fatal error" remainingTxns=0

S3 bucket is populated with files and retrieving also works - it's just amount of errors is insane

We're running EC2 instances, is there a way we can find why are we getting Bad Gateway errors in Splunk and how to solve them?

Thx, Serge.

Labels (1)
0 Karma

srajarat2
Path Finder

Splunk Support did confirm that SmartStore is only supported on Linux at this time and not on Windows.

0 Karma

srajarat2
Path Finder

Seems like Splunk SmartStore is not supported on Windows, yet?

Check this link.
https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/SmartStoresystemrequirements#Operating_sy...

It refers to Linux only.

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

With Splunk smartstore Generally, ignore HTTP status code on their own

  • There could be intermittent problems in these codes which get resolved by a retry.
  • The codes become valuable when investigating other issues.
  • They could raise red flags if some HTTP status codes start appearing, such as 401's, for example. It wouldn't be practical to enumerate which codes and with which frequency indicates a problem.
0 Karma

thormanrd
Path Finder

There is fundamentally a problem ignoring HTTP responses.  Many issues are illuminated with these, such as:

  1. HTTP 503 Slow down - typically a keyspace design issue
  2. HTTP 502 Read timeout - long reads and head operations with too little timeout setting
  3. HTTP 502 Connection timeout - long or busy network connections with too little timeout setting
  4. HTTP 500 Internal server error - poor exception processing that needs bug report
  5. HTTP 502 Name or Service not known - failure in DNS services
  6. HTTP 502 Partial upload - failure in multi-part upload operation that needs bug report
0 Karma

skalliger
SplunkTrust
SplunkTrust

Always try to keep the Windows path as short as possible, e.g. install to E:\splunk.

There is a permission error in your event provided. Does your splunk user have admin permissions? That's usually needed for the Program Files folder.

Skalli

0 Karma

serge_ohpen
New Member

The permissions are correct but just in case I have reset them yet still getting access denied messages. What is strange, I can see .tmp files been created there with 0 size and then disappear again.

Thx, Serge.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...