Internal server Error when exposing 8089 port for splunkd rest API with an ingress.

shail — Mon, 09 Dec 2024 17:22:32 GMT

I have been trying to set up splunk on my Kubernetes cluster so i can use it with a python script to access the rest API.

i have a splunk enterprise standalone instance running.

i used traefik ingress to expose port 8089

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: splunk-ingress namespace: splunk annotations: cert-manager.io/cluster-issuer: letsencrypt-issuer traefik.ingress.kubernetes.io/router.entrypoints: websecure spec: ingressClassName: common-traefik tls: - hosts: - splunk.example.com secretName: app-certificate rules: - host: splunk.example.com http: paths: - path: / pathType: Prefix backend: service: name: splunk-stdln-standalone-service port: number: 8089

when i try to curl to the client it returns internal server error

curl -X POST https://splunk.example.com/services/auth/login --data-urlencode username=admin --data-urlencode password=<mysplunkpassword> -k -v

output:

* Host splunk.example.com:443 was resolved. * IPv6: (none) * IPv4: xx.xx.xxx.xxx * Trying xx.xx.xxx.xxx:443... * Connected to splunk.example.com (xx.xx.xxx.xxx) port 443 * ALPN: curl offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF * ALPN: server accepted h2 * Server certificate: * subject: CN=splunk.example.com * start date: Dec 6 23:53:06 2024 GMT * expire date: Mar 6 23:53:05 2025 GMT * issuer: C=US; O=Let's Encrypt; CN=R10 * SSL certificate verify ok. * using HTTP/2 * [HTTP/2] [1] OPENED stream for https://splunk.example.com/services/auth/login * [HTTP/2] [1] [:method: POST] * [HTTP/2] [1] [:scheme: https] * [HTTP/2] [1] [:authority: splunk.example.com] * [HTTP/2] [1] [:path: /services/auth/login] * [HTTP/2] [1] [user-agent: curl/8.7.1] * [HTTP/2] [1] [accept: */*] * [HTTP/2] [1] [content-length: 34] * [HTTP/2] [1] [content-type: application/x-www-form-urlencoded] > POST /services/auth/login HTTP/2 > Host: splunk.example.com > User-Agent: curl/8.7.1 > Accept: */* > Content-Length: 34 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 34 bytes < HTTP/2 500 < content-length: 21 < date: Mon, 09 Dec 2024 06:54:50 GMT < * Connection #0 to host splunk.example.com left intact Internal Server Error%

when i port forward to localhost the curl works

curl -X POST https://localhost:8089/services/auth/login --data-urlencode username=admin --data-urlencode password=<mysplunkpassword> -k -v

output:

Note: Unnecessary use of -X or --request, POST is already inferred. * Host localhost:8089 was resolved. * IPv6: ::1 * IPv4: 127.0.0.1 * Trying [::1]:8089... * Connected to localhost (::1) port 8089 * ALPN: curl offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): * (304) (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / [blank] / UNDEF * ALPN: server did not agree on a protocol. Uses default. * Server certificate: * subject: CN=SplunkServerDefaultCert; O=SplunkUser * start date: Dec 9 02:21:04 2024 GMT * expire date: Dec 9 02:21:04 2027 GMT * issuer: C=US; ST=CA; L=San Francisco; O=Splunk; CN=SplunkCommonCA; emailAddress=support@splunk.com * SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway. * using HTTP/1.x > POST /services/auth/login HTTP/1.1 > Host: localhost:8089 > User-Agent: curl/8.7.1 > Accept: */* > Content-Length: 34 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 34 bytes < HTTP/1.1 200 OK < Date: Mon, 09 Dec 2024 06:59:54 GMT < Expires: Thu, 26 Oct 1978 00:00:00 GMT < Cache-Control: no-store, no-cache, must-revalidate, max-age=0 < Content-Type: text/xml; charset=UTF-8 < X-Content-Type-Options: nosniff < Content-Length: 204 < Connection: Keep-Alive < X-Frame-Options: SAMEORIGIN < Server: Splunkd < <response> <sessionKey> {some sessionKey...} </sessionKey> <messages> <msg code=""></msg> </messages> </response> * Connection #0 to host localhost left intact

I am using default confs
not sure if i need to update my server.conf for this

more context:
i checked the splunkd.log from when i made the request
and i get these logs:

12-09-2024 17:19:36.904 +0000 WARN SSLCommon [951 HTTPDispatch] - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='bad certificate'. 12-09-2024 17:19:36.904 +0000 WARN HttpListener [951 HTTPDispatch] - Socket error from 192.168.xx.xx:52528 while idling: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Re: Internal server Error when exposing 8089 port for splunkd rest API with an ingress.

PickleRick — Mon, 09 Dec 2024 10:11:56 GMT

Let me ask you first, why would you want to map your 8089 splunkd port to 443? 443 is for webUI (if enabled and redirected from the default 8000). 8089 is the port your API is expected to be at.

Re: Internal server Error when exposing 8089 port for splunkd rest API with an ingress.

shail — Mon, 09 Dec 2024 17:11:16 GMT

my goal was to test splunk Rest API, Since I just needed to create an endpoint to access it
so i used the hostname directly. I dont need to use the webUI

Does this affect the splunk configuration?
I am not sure what the issue is here or why would i get an internal server error?
Any hints appreciated!

topic Internal server Error when exposing 8089 port for splunkd rest API with an ingress. in Splunk Enterprise

Internal server Error when exposing 8089 port for splunkd rest API with an ingress.

Re: Internal server Error when exposing 8089 port for splunkd rest API with an ingress.

Re: Internal server Error when exposing 8089 port for splunkd rest API with an ingress.