unable to access Splunk web GUI via (AWS) Elastic Load Balancing (ELB) & Splunk is not indexing the kinesis data

[ktiriveedi]

Our earlier set up is , We are pulling the data from cloudwtach to Splunk. There we are getting throttling issues.

To avoid throttling issues we are implementing the kinesis firehose implementation.

For your reference, Below is the new approach diagram  

We are trying  sending cloudwatch logs to kinesis firehose, from kinesis firehose to the splunk system.

Kinesis firehose is able to send the data to splunk but it was not indexing in the splunk.

So far I have done this workaround 

Workaround and issues:

where we are unable to access Splunk web GUI using Amazon Web Services (AWS) Elastic Load Balancing (ELB) DNS name example:- http://ELB:PORT or https://ELB/en-US/account/login.

I mapped elb dns name with domain name - still we are getting the same issue.

Details:- Classic Load balancer and Splunk enterprise 7.2.0 version free license & Stand alone

We are able to connect port 8000 when trying with http;//IP:8000 

we were able to open GUI using the ELB name on HTTP load balancer port 8000 & Instance level port 8000.

We are using ACM certificate and attached to ELB

Health Checks are passed and Instances is InService 

Health Check settings Ping protocol: HTTPS

Ping port: 8088

Ping path: HTTPS:8088/services/collector/health/1.0

Timeout: 5 seconds

Interval: 30 seconds

Unhealthy threshold: 2

Healthy threshold: 10

We deployed in VPC, enabled network Security group rules internally between ELB and EC2 instances.

Below are the ELB configurations:-

Health Check settings Ping protocol: HTTPS

Ping port: 8088

Ping path: HTTPS:8088/services/collector/health/1.0

Timeout: 5 seconds

Interval: 30 seconds

Unhealthy threshold: 2

Healthy threshold: 10

Listeners:- ELB -HTTP -8000- Instance protocol - HTTP -8000 - We are able to access the UI

Listeners:- ELB -HTTP -443- Instance protocol - HTTP -8000 - Not working

-> Also  tested by on enabling the HTTPS on web.conf

enableSplunkWebSSL = true, Able to open GUI on Https://IP:PORT but not Https://ELB:PORT

As per Splunk previous answers on the same issues, we did verify web.conf under /splunk_home/splunk/etc/system/default/web.conf for TLS1.2 version cipherSuite.

it exists in our splunk web.conf default path

Few more questions:

Which port we need to give in HEC endpoint 8088 or 8000 or 443 ?

In ELB listeners what are loadbalancer port & Instance port ?

Note: I am using splunk enterprise free license. 

[maraman_splunk]

web port is 8000 (dont leave that open to Internet, you have no auth with the free license !)

hec port is 8088

use this to test hec

launch this from browser

https://hec-splunk.mydomain.com:8088/services/collector/health/1.0

by replacing hec-splunk.mydomain.com with the public dns name you use from the ELB that does HEC

you should have a valid certificate on the elb or Kinesis Firehose will refuse to connect (generating one is free via ACM for ELB)

[ktiriveedi]

We have ACM certificate already in place.

I launched from browser https://elbdnsname:8088/services/collector/health/1.0 ,

I am getting the page isn't working, HTTP Error 504 

In ELB listeners Load balancer port - 8088 & Instance -port -8088 pointed to ACM certificate.

I have updated  https://elbdnsname:8088/services/collector/health/1.0 in kinesis also. I am getting below error

Could not connect to the HEC endpoint. The host does not match the certificate provided by the peer. Make sure that the certificate and the host are valid

Error code & Version - 

Splunk.SSLUnverified

6