Knowledge Management

Experiencing [SmartStore] HTTP 502 network errors with S3?

serge_ohpen
New Member

Hello All,

After configuring migration for a few indexes, the following errors is filling up the log on all cluster peers:

06-27-2019 07:43:11.914 +0200 ERROR S3Client - command=get transactionId=0x4053369110 rTxnId=0x4051be42d0 status=comple
ted success=N localPath="D:\Program Files\Splunk\var\lib\splunk\db\db_1561365505_1561358169_598_A3573547-8422-44
29-B22D-4867A0A1B8E8.tmp" offset=0 error="can not open file" reason="Access is denied."
06-27-2019 07:43:11.915 +0200 ERROR RetryableClientTransaction - transactionDone(): groupId=0x4043b50ef0 rTxnId=0x4051b
e42d0 transactionId=0x4053369110 success=N HTTP-statusCode=502 HTTP-statusDescription="network error" retries=429496729
5 retry=N no_retry_reason="transaction group had fatal error" remainingTxns=0

S3 bucket is populated with files and retrieving also works - it's just amount of errors is insane

We're running EC2 instances, is there a way we can find why are we getting Bad Gateway errors in Splunk and how to solve them?

Thx, Serge.

Labels (1)
0 Karma

srajarat2
Path Finder

Splunk Support did confirm that SmartStore is only supported on Linux at this time and not on Windows.

0 Karma

srajarat2
Path Finder

Seems like Splunk SmartStore is not supported on Windows, yet?

Check this link.
https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/SmartStoresystemrequirements#Operating_sy...

It refers to Linux only.

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

With Splunk smartstore Generally, ignore HTTP status code on their own

  • There could be intermittent problems in these codes which get resolved by a retry.
  • The codes become valuable when investigating other issues.
  • They could raise red flags if some HTTP status codes start appearing, such as 401's, for example. It wouldn't be practical to enumerate which codes and with which frequency indicates a problem.
0 Karma

thormanrd
Path Finder

There is fundamentally a problem ignoring HTTP responses.  Many issues are illuminated with these, such as:

  1. HTTP 503 Slow down - typically a keyspace design issue
  2. HTTP 502 Read timeout - long reads and head operations with too little timeout setting
  3. HTTP 502 Connection timeout - long or busy network connections with too little timeout setting
  4. HTTP 500 Internal server error - poor exception processing that needs bug report
  5. HTTP 502 Name or Service not known - failure in DNS services
  6. HTTP 502 Partial upload - failure in multi-part upload operation that needs bug report
0 Karma

skalliger
Motivator

Always try to keep the Windows path as short as possible, e.g. install to E:\splunk.

There is a permission error in your event provided. Does your splunk user have admin permissions? That's usually needed for the Program Files folder.

Skalli

0 Karma

serge_ohpen
New Member

The permissions are correct but just in case I have reset them yet still getting access denied messages. What is strange, I can see .tmp files been created there with 0 size and then disappear again.

Thx, Serge.

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...