Consume API of a remote hosted splunk

[tbhasme]

I have a react app running locally and I need to consume APIs of Splunk which is hosted on some other server. In order to talk to the that server, here are the things I did:
1. Updated splunk with my own signed certificates and installed the CA vert on my machine and browser (as listed on https://docs.splunk.com/Documentation/Splunk/7.3.0/Security/Howtoself-signcertificates, https://docs.splunk.com/Documentation/Splunk/7.3.0/Security/HowtoprepareyoursignedcertificatesforSpl..., and https://docs.splunk.com/Documentation/Splunk/7.3.0/Security/ConfigureSplunkforwardingtousesignedcert...)
2. set crossOriginSharingPolicy = *

Here is my client code:

var details = {
            'username': 'admin',
            'password': '<pwd>'
        };
        var formBody = [];
        for (var property in details) {
          var encodedKey = encodeURIComponent(property);
          var encodedValue = encodeURIComponent(details[property]);
          formBody.push(encodedKey + "=" + encodedValue);
        }
        formBody = formBody.join("&");
        console.log(formBody)
        fetch("https://localhost:8089/services/auth/login", {
              method: "POST",
              body: formBody,                  
              headers: { 'Content-type': 'application/x-www-form-urlencoded',
             }

            })
              .then(res => {
                if (res.ok) {
                    console.log("Response json data. -->" + res)
                  return res;
                } else {
                  throw Error(res.statusText);
                }
              })
              .then(json => {
                this.setState({
                  token: json
                });
              })
              .catch(error => console.error(error));

Splunk is hosted on a remote server which I am accessing through port forwarding.

With this setup,
1. On Chrome: I am getting ERR_CERT_COMMON_NAME_INVALID. It seems like CN mismatch but I really don't think that is the issue.
2. On FIrefox: I can see the session token in web console but in code, i get a response object of type CORS.

I am not really sure where have I gone wrong.

[tbhasme · Accepted solution]

ok.. so turns out, changing the way to call the api from UI needed a different approach (not sure why)... right now only additional setting in the splunk server I have is

[sslConfig]
enableSplunkdSSL = false
.
.
.
[httpServer]
crossOriginSharingPolicy = *

and ui code is now using async method to fetch the token:

getToken = async() => {

        var details = {
          'username': 'admin',
          'password': 'J17kP3xjyBSvvKSuR5HrEYSI8w4'
      };

      var formBody = [];
      for (var property in details) {
        var encodedKey = encodeURIComponent(property);
        var encodedValue = encodeURIComponent(details[property]);
        formBody.push(encodedKey + "=" + encodedValue);
      }
      formBody = formBody.join("&");
      console.log(formBody)

        const response = await fetch("http://splunk.local:8089/services/auth/login?output_mode=json", {
          method: "POST",
          body: formBody,
          headers: { 
              'Content-type': 'application/x-www-form-urlencoded' 
            }
        });
        const responseInJson = await response.json();
        console.log(`Session Key - ${responseInJson.sessionKey}`); 
        return responseInJson.sessionKey
      }

And it somehow now seems to work!

The same server settings do not work with the earlier UI code but work with async somehow!

[jkat54]

Of course that's the issue. You're proxying through localhost to get to a remote sever with a different CN.

You could disable ssl cert validation in your app or directly connect to the server using its proper CN/FQDN

[tbhasme]

I have also tried by disabling certificate validation in Chrome but that result in an empty response of type cors.

[jkat54]

It's just not going to work this way. Either fetch the splunk endpoint by its fqdn/cn (and be sure it's CA is in your trusted root store) or find another way to do what you want.

[tbhasme]

It is not possible to access splunk using its fqdn as it is in some other network only accessible through ssh.
I have already configured Splunk with a CA whose CN I have kept '*.local'. The server certificate is also generated with same CN. That CA is also installed in my machine as well as Chrome. I am then running my local app as http://splunk.local:3000 which hits splunk on https://splunk.local:8089.
It still says ERR_CERT_COMMON_NAME_INVALID.

However, if this is a cert issue, it should have worked with ssl disabled. When I tried by disabling splunkd's ssl, I again started getting response type of cors.

[jkat54]

Have you seen the "hosts" file before?

Put the proper IP and fqdn in your hosts file and enjoy.

Another issue might be that the cert you think you have installed on the splunk server is not the correct one. Try using

openssl s_client -connect yourserver:8089 

And share the print out.

[tbhasme]

Yes, I have already made the entry (127.0.0.1 splunk.local). That's how I am accessing splunk.local.

Here is the output:

 openssl s_client -connect splunk.local:8089
CONNECTED(00000003)
depth=0 C = AU, ST = Some-State, O = Splunk, CN = *.local
verify error:num=18:self signed certificate
verify return:1
depth=0 C = AU, ST = Some-State, O = Splunk, CN = *.local
verify return:1
---
Certificate chain
 0 s:/C=AU/ST=Some-State/O=Splunk/CN=*.local
   i:/C=AU/ST=Some-State/O=Splunk/CN=*.local
 1 s:/C=AU/ST=Some-State/O=Splunk/CN=*.local
   i:/C=AU/ST=Some-State/O=Splunk/CN=*.local
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDBjCCAe4CCQDgMbcsg7UyGTANBgkqhkiG9w0BAQsFADBFMQswCQYDVQQGEwJB
VTETMBEGA1UECAwKU29tZS1TdGF0ZTEPMA0GA1UECgwGU3BsdW5rMRAwDgYDVQQD
DAcqLmxvY2FsMB4XDTE5MDcwNTE3MTkwMVoXDTIyMDcwNDE3MTkwMVowRTELMAkG
A1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxDzANBgNVBAoMBlNwbHVuazEQ
MA4GA1UEAwwHKi5sb2NhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AM0j+3TbxoOwbHhDc6gF/stgNnUr3IAbZ8Yw2UJued32lrtHD6yGq9ESQ6vBNbFC
AwUpxoItc9/EX7b6Ct7uJOpWVNOd54CosuVkZksqIAyrI0lZ7OLqkH2yUseSWCIA
GRkCfxSTzmr3oPyId/ifuWvQcdMPDF4Af4XNLycSUo3vv+RZXf8oTnJY4+0vKXuy
JonxsilcKhVkUAOmyH7U9rK1Z041Hnlsx7FHIwmcf5U4ThLtyJmSOHcpH/ZA8gxt
9OrxaDV6RnltSzXfrE9E+QXcn7K7m0t0aJcJ14y7ccsoGJsr+MrHOz5OQ7WWAF8q
FPcVEA/sWL+wRjkOK2dMtosCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAh8Cgv29v
Y+J0HRsrSNGLn5J8z/vU1vna0K14LvsyWLfa6W9Wif8kKNgTplSi3FJZnlmHHttL
gKQoxJAq6qQ5/Z2EgDUqUJf7LuAeVzCYbN3fbf3lFvYtOQv438j0ckPXsSwpQU+i
vI+ry3FTa4KqpCqzpsso9JkkzHhCqhIRJOS0inxPkQLVmHvxbFjMYCLRwEM/wfaL
QbfV9tpzXKbiJpkWdkMe6AyglNar/FU50lkUm7dTVdjvrW7ZJ6zN4l6tRpMrBJcc
2HS+lRCeifvPH8wq0XVnNUThTaJU91DzyrN1GqXQqduhe7FsL4dqMYT7Cqzm0neR
RAgDoFQpXWPZcQ==
-----END CERTIFICATE-----
subject=/C=AU/ST=Some-State/O=Splunk/CN=*.local
issuer=/C=AU/ST=Some-State/O=Splunk/CN=*.local
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2250 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: A71E3F0100E72CA2EBD8C4E2B38148A6BD8D1BDF7CB59E684C71A27B171BE54D
    Session-ID-ctx: 
    Master-Key: A96AA2D016D688AA631158C1E10D80CEE87C303C521E207D458A8484E3C04E4EB09216E1B4FB02FDD3A0C19AD81B43C4
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 41 4c a2 39 1e 22 44 4b-4c 97 c4 d5 49 19 f8 dc   AL.9."DKL...I...
    0010 - 95 07 72 3c 33 fd 80 ef-6e 67 4f d2 3d 99 a6 f4   ..r<3...ngO.=...
    0020 - 08 cd 06 aa 56 ed 6d 13-28 84 d1 e3 8d 45 1e 40   ....V.m.(....E.@
    0030 - ad 89 84 6d bf 80 ee 49-e0 4c 99 50 61 22 7b 23   ...m...I.L.Pa"{#
    0040 - 83 a4 03 7d 13 c7 1c 38-a7 86 82 fc b8 66 1e 22   ...}...8.....f."
    0050 - df a7 28 ae d1 b6 17 4e-fa 8c ba 46 44 e5 25 a3   ..(....N...FD.%.
    0060 - 21 b8 a8 5f b8 39 c0 7c-6f f8 5d 38 9a df 02 d9   !.._.9.|o.]8....
    0070 - 77 65 a9 32 d8 df 9b 84-3a 07 de 40 e4 6e 15 d6   we.2....:..@.n..
    0080 - 29 16 b4 b9 76 2c c2 5b-85 07 12 4c 32 18 2b d6   )...v,.[...L2.+.
    0090 - b0 d6 4e f6 ae e5 04 68-54 47 cc ce 9b 1c 71 7b   ..N....hTG....q{
    00a0 - a7 e4 73 01 44 2e b6 3d-52 fc 88 dc 45 c6 b7 5e   ..s.D..=R...E..^
    00b0 - bf 4e 10 97 ee ae 71 7c-33 41 54 62 46 20 85 86   .N....q|3ATbF ..

    Start Time: 1562352257
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---
read:errno=0

[tbhasme]

Now, I just opened the url https://splunk.local:8089/services/auth/login (which was showing the CN error) in another tab which showed me security risk. I accepted it and refreshed the page. Now there is no error but again, the response is empty and of type cors.

[jkat54]

Did you try putting this certificate in your trusted root store?

[tbhasme]

Yes, I installed the certificate in browser as well as system's trusted root store. I did restart browser and even incognito mode.

[jkat54]

Check this out, you need to set the CORS policy in your request headers.

https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

[jkat54]

I might be wrong there...

You did this already right? And restarted splunk? Make sure no typos please:

If you are using CORS, you'll need to configure the Splunk server to allow your web site to communicate with it using a CORS connection. To do this, add your web site's address as a trusted HTTP origin to the crossOriginSharingPolicy attribute in the server.conf configuration file.

For example, add this stanza to the $SPLUNK_HOME/etc/system/local/server.conf configuration file, then restart Splunk:

[httpServer]
crossOriginSharingPolicy = your_site_address

[tbhasme]

Yes, that was the second step of my setup... and yes, I did restart splunkd service...

[jkat54]

Can you share the print out from this command?

 $splunk_home/bin/splunk btool server list httpServer  --debug

[tbhasme]

/opt/splunk/etc/system/local/server.conf   [httpServer]
/opt/splunk/etc/system/default/server.conf acceptFrom = *
/opt/splunk/etc/system/default/server.conf allowBasicAuth = true
/opt/splunk/etc/system/default/server.conf allowCookieAuth = true
/opt/splunk/etc/system/default/server.conf allowEmbedTokenAuth = true
/opt/splunk/etc/system/default/server.conf atomFeedStylesheet = /static/atom.xsl
/opt/splunk/etc/system/default/server.conf basicAuthRealm = /splunk
/opt/splunk/etc/system/default/server.conf busyKeepAliveIdleTimeout = 12
/opt/splunk/etc/system/default/server.conf cookieAuthHttpOnly = true
/opt/splunk/etc/system/default/server.conf cookieAuthSecure = true
/opt/splunk/etc/system/local/server.conf   crossOriginSharingPolicy = *
/opt/splunk/etc/system/default/server.conf dedicatedIoThreads = 0
/opt/splunk/etc/system/default/server.conf follow-symlinks = false
/opt/splunk/etc/system/default/server.conf forceHttp10 = auto
/opt/splunk/etc/system/default/server.conf keepAliveIdleTimeout = 7200
/opt/splunk/etc/system/default/server.conf max-age = 3600
/opt/splunk/etc/system/default/server.conf maxSockets = 0
/opt/splunk/etc/system/default/server.conf maxThreads = 0
/opt/splunk/etc/system/default/server.conf max_content_length = 2147483648
/opt/splunk/etc/system/default/server.conf streamInWriteTimeout = 5
/opt/splunk/etc/system/default/server.conf x_frame_options_sameorigin = true

[jkat54]

Doesn't make much sense then does it? You should probably open a ticket.

[jkat54]

What version of splunk are you using?

Afaik this will only work if you have splunk enterprise.

[jkat54]

And yes, when working with certificates like this, you'll want to completely close and open your browser between tests.