Our earlier set up is , We are pulling the data from cloudwtach to Splunk. There we are getting throttling issues.
To avoid throttling issues we are implementing the kinesis firehose implementation.
For your reference, Below is the new approach diagram
We are trying sending cloudwatch logs to kinesis firehose, from kinesis firehose to the splunk system.
Kinesis firehose is able to send the data to splunk but it was not indexing in the splunk.
So far I have done this workaround
Workaround and issues:
where we are unable to access Splunk web GUI using Amazon Web Services (AWS) Elastic Load Balancing (ELB) DNS name example:- http://ELB:PORT or https://ELB/en-US/account/login.
I mapped elb dns name with domain name - still we are getting the same issue.
Details:- Classic Load balancer and Splunk enterprise 7.2.0 version free license & Stand alone
We are able to connect port 8000 when trying with http;//IP:8000
we were able to open GUI using the ELB name on HTTP load balancer port 8000 & Instance level port 8000.
We are using ACM certificate and attached to ELB
Health Checks are passed and Instances is InService
Health Check settings Ping protocol: HTTPS
Ping port: 8088
Ping path: HTTPS:8088/services/collector/health/1.0
Timeout: 5 seconds
Interval: 30 seconds
Unhealthy threshold: 2
Healthy threshold: 10
We deployed in VPC, enabled network Security group rules internally between ELB and EC2 instances.
Below are the ELB configurations:-
Listeners:- ELB -HTTP -8000- Instance protocol - HTTP -8000 - We are able to access the UI
Listeners:- ELB -HTTP -443- Instance protocol - HTTP -8000 - Not working
-> Also tested by on enabling the HTTPS on web.conf
enableSplunkWebSSL = true, Able to open GUI on Https://IP:PORT but not Https://ELB:PORT
As per Splunk previous answers on the same issues, we did verify web.conf under /splunk_home/splunk/etc/system/default/web.conf for TLS1.2 version cipherSuite.
it exists in our splunk web.conf default path
Few more questions:
Which port we need to give in HEC endpoint 8088 or 8000 or 443 ?
In ELB listeners what are loadbalancer port & Instance port ?
Note: I am using splunk enterprise free license.
web port is 8000 (dont leave that open to Internet, you have no auth with the free license !)
hec port is 8088
use this to test hec
launch this from browser
by replacing hec-splunk.mydomain.com with the public dns name you use from the ELB that does HEC
you should have a valid certificate on the elb or Kinesis Firehose will refuse to connect (generating one is free via ACM for ELB)
We have ACM certificate already in place.
I launched from browser https://elbdnsname:8088/services/collector/health/1.0 ,
I am getting the page isn't working, HTTP Error 504
In ELB listeners Load balancer port - 8088 & Instance -port -8088 pointed to ACM certificate.
I have updated https://elbdnsname:8088/services/collector/health/1.0 in kinesis also. I am getting below error
Could not connect to the HEC endpoint. The host does not match the certificate provided by the peer. Make sure that the certificate and the host are valid
Error code & Version -