Stream unable to ping status=502

[johnansett]

Hello Splunkers,

This might just be a sanity check....but I'll ask anyway.

Deployed Stream yesterday to around ~360 hosts (all Windows - need to run the setpermissions on the nix before they'll come up).  Everything set to estimate except DNS which is rolled out to all hosts.

I can see a nice, constant flow of data into the stream index, ES is triggering notables, everything seems like it is working nicely.

I check out the Stream forwarder status and I'm bouncing around from 100-300 hosts with an error status over the last hour (this is constant since I've deployed) a fairly constant active of between 50-80 and a couple in a warning.

When I check out the internal logs I see this:

Unable to ping server (8f938d78-0c1b-43a6-b32c-e6e094e7bc2b): /en-us/custom/splunk_app_stream/ping/ status=502

Checked we can ping that.  Also, check and these same hosts have data, in fact ALL hosts have data.

As I look in the Stream SH I see these corrosponding errors

10-31-2020 16:21:51.375 +0000 ERROR HttpClientRequest - HTTP client error=Read Timeout while accessing server=http://127.0.0.1:8065 for request=http://127.0.0.1:8065/en-us/custom/splunk_app_stream/ping/.

I'm thinking this is a resource issue on the Stream SH, but looking at the stats in MC everything is fine - 30% CPU.

The Stream SH is also the monitoring console and not a big box - 8 cores/16GB ram, RHEL8.  It's doing nothing other than Stream and MC and no one is using this.

Are there any limits that might cause this?

The 502 seems to me to indicate the MC/Stream server is the cause, but before I throw more cores to see it would be nice to confirm. 

Any help appreciated!

Cheers!

[inventsekar]

Hi @johnansett may we know 

1. if you installed "Splunk TA for Stream" or "Splunk App for Stream" (or both?!)

2. For ping error 502 Google gives me - "A 502 Bad Gateway indicates that the edge server (server acting as a proxy) was not able to get a valid or any response from the origin server (also called upstream server)"-- -does it ring a bell?

3. please check these similar posts:

this post says some inputs.conf update - 

https://community.splunk.com/t5/All-Apps-and-Add-ons/Unable-to-ping-server-message-in-streadfwd-log/...

https://community.splunk.com/t5/Archive/Splunk-Stream-not-working/m-p/476237

Best Regards,

Sekar

PS - Karma points appreciated!

[johnansett]

Both are installed.  502 is a standard server error, I'm seeing errors on the server too - see the original post.  I confirmed server listening on 8065.

The posts aren't relevant, looked through answers and I have inputs setup

It's working despite these errors, but the server has horrible performance which is the biggest issue, I doubled CPU/RAM (16 cores / 64gb) but this didn't help.