Internal server Error when exposing 8089 port for splunkd rest API with an ingress.

[shail]

I have been trying to set up splunk on my Kubernetes cluster so i can use it with a python script to access the rest API.

i have a splunk enterprise standalone instance running.

i used traefik ingress to expose port 8089 

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: splunk-ingress
  namespace: splunk
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-issuer
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  ingressClassName: common-traefik
  tls:
    - hosts:
        - splunk.example.com
      secretName: app-certificate
  rules:
    - host: splunk.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: splunk-stdln-standalone-service
                port:
                  number: 8089

when i try to curl to the client it returns internal server error

curl -X POST https://splunk.example.com/services/auth/login --data-urlencode username=admin --data-urlencode password=<mysplunkpassword> -k -v

output:

* Host splunk.example.com:443 was resolved.
* IPv6: (none)
* IPv4: xx.xx.xxx.xxx
*   Trying xx.xx.xxx.xxx:443...
* Connected to splunk.example.com (xx.xx.xxx.xxx) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=splunk.example.com
*  start date: Dec  6 23:53:06 2024 GMT
*  expire date: Mar  6 23:53:05 2025 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://splunk.example.com/services/auth/login
* [HTTP/2] [1] [:method: POST]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: splunk.example.com]
* [HTTP/2] [1] [:path: /services/auth/login]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 34]
* [HTTP/2] [1] [content-type: application/x-www-form-urlencoded]
> POST /services/auth/login HTTP/2
> Host: splunk.example.com
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 34
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 34 bytes
< HTTP/2 500 
< content-length: 21
< date: Mon, 09 Dec 2024 06:54:50 GMT
< 
* Connection #0 to host splunk.example.com left intact
Internal Server Error%   

when i port forward to localhost the curl works

curl -X POST https://localhost:8089/services/auth/login --data-urlencode username=admin --data-urlencode password=<mysplunkpassword> -k -v

output:

Note: Unnecessary use of -X or --request, POST is already inferred.
* Host localhost:8089 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8089...
* Connected to localhost (::1) port 8089
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=SplunkServerDefaultCert; O=SplunkUser
*  start date: Dec  9 02:21:04 2024 GMT
*  expire date: Dec  9 02:21:04 2027 GMT
*  issuer: C=US; ST=CA; L=San Francisco; O=Splunk; CN=SplunkCommonCA; emailAddress=support@splunk.com
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* using HTTP/1.x
> POST /services/auth/login HTTP/1.1
> Host: localhost:8089
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 34
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 34 bytes
< HTTP/1.1 200 OK
< Date: Mon, 09 Dec 2024 06:59:54 GMT
< Expires: Thu, 26 Oct 1978 00:00:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, max-age=0
< Content-Type: text/xml; charset=UTF-8
< X-Content-Type-Options: nosniff
< Content-Length: 204
< Connection: Keep-Alive
< X-Frame-Options: SAMEORIGIN
< Server: Splunkd
< 
<response>
  <sessionKey> {some sessionKey...} </sessionKey>
  <messages>
    <msg code=""></msg>
  </messages>
</response>
* Connection #0 to host localhost left intact

 I am using default confs
not sure if i need to update my server.conf  for this

more context:
i checked the splunkd.log from when i made the request
and i get these logs:

12-09-2024 17:19:36.904 +0000 WARN  SSLCommon [951 HTTPDispatch] - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='bad certificate'.

12-09-2024 17:19:36.904 +0000 WARN  HttpListener [951 HTTPDispatch] - Socket error from 192.168.xx.xx:52528 while idling: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

[PickleRick]

Let me ask you first, why would you want to map your 8089 splunkd port to 443? 443 is for webUI (if enabled and redirected from the default 8000). 8089 is the port your API is expected to be at.

[shail]

my goal was to test splunk Rest API, Since I just needed to create an endpoint to access it
so i used the hostname directly. I dont need to use the webUI 

Does this affect the splunk configuration?
I am not sure what the issue is here or why would i get an internal server error?
Any hints appreciated!