I have been trying to set up splunk on my Kubernetes cluster so i can use it with a python script to access the rest API. i have a splunk enterprise standalone instance running. i used traefik ingress to expose port 8089 apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: splunk-ingress
namespace: splunk
annotations:
cert-manager.io/cluster-issuer: letsencrypt-issuer
traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
ingressClassName: common-traefik
tls:
- hosts:
- splunk.example.com
secretName: app-certificate
rules:
- host: splunk.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: splunk-stdln-standalone-service
port:
number: 8089 when i try to curl to the client it returns internal server error curl -X POST https://splunk.example.com/services/auth/login --data-urlencode username=admin --data-urlencode password=<mysplunkpassword> -k -v output: * Host splunk.example.com:443 was resolved.
* IPv6: (none)
* IPv4: xx.xx.xxx.xxx
* Trying xx.xx.xxx.xxx:443...
* Connected to splunk.example.com (xx.xx.xxx.xxx) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
* subject: CN=splunk.example.com
* start date: Dec 6 23:53:06 2024 GMT
* expire date: Mar 6 23:53:05 2025 GMT
* issuer: C=US; O=Let's Encrypt; CN=R10
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://splunk.example.com/services/auth/login
* [HTTP/2] [1] [:method: POST]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: splunk.example.com]
* [HTTP/2] [1] [:path: /services/auth/login]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 34]
* [HTTP/2] [1] [content-type: application/x-www-form-urlencoded]
> POST /services/auth/login HTTP/2
> Host: splunk.example.com
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 34
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 34 bytes
< HTTP/2 500
< content-length: 21
< date: Mon, 09 Dec 2024 06:54:50 GMT
<
* Connection #0 to host splunk.example.com left intact
Internal Server Error% when i port forward to localhost the curl works curl -X POST https://localhost:8089/services/auth/login --data-urlencode username=admin --data-urlencode password=<mysplunkpassword> -k -v output: Note: Unnecessary use of -X or --request, POST is already inferred.
* Host localhost:8089 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:8089...
* Connected to localhost (::1) port 8089
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
* subject: CN=SplunkServerDefaultCert; O=SplunkUser
* start date: Dec 9 02:21:04 2024 GMT
* expire date: Dec 9 02:21:04 2027 GMT
* issuer: C=US; ST=CA; L=San Francisco; O=Splunk; CN=SplunkCommonCA; emailAddress=support@splunk.com
* SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* using HTTP/1.x
> POST /services/auth/login HTTP/1.1
> Host: localhost:8089
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 34
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 34 bytes
< HTTP/1.1 200 OK
< Date: Mon, 09 Dec 2024 06:59:54 GMT
< Expires: Thu, 26 Oct 1978 00:00:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, max-age=0
< Content-Type: text/xml; charset=UTF-8
< X-Content-Type-Options: nosniff
< Content-Length: 204
< Connection: Keep-Alive
< X-Frame-Options: SAMEORIGIN
< Server: Splunkd
<
<response>
<sessionKey> {some sessionKey...} </sessionKey>
<messages>
<msg code=""></msg>
</messages>
</response>
* Connection #0 to host localhost left intact I am using default confs not sure if i need to update my server.conf for this more context: i checked the splunkd.log from when i made the request and i get these logs: 12-09-2024 17:19:36.904 +0000 WARN SSLCommon [951 HTTPDispatch] - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='bad certificate'.
12-09-2024 17:19:36.904 +0000 WARN HttpListener [951 HTTPDispatch] - Socket error from 192.168.xx.xx:52528 while idling: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
... View more