OpenShift Log Forwarding to Splunk?

[Suchita1]

We are using OpenShift 4.11.27 and now looking for OpenShift Log Forwarding to Splunk.

Did below changes at OpenShift end to configure splunk:

1. Installed cluster-logging and elasticsearch-operator into OpenShift.

   $ oc get csv -n openshift-logging

   NAME                                     DISPLAY                            VERSION   REPLACES                                       PHASE

   cluster-logging.v5.6.3                   Red Hat OpenShift Logging          5.6.3     cluster-logging.v5.6.2                         Succeeded

   elasticsearch-operator.v5.6.3            OpenShift Elasticsearch Operator   5.6.3     elasticsearch-operator.v5.6.2                  Succeeded

   1. Created secret vector-splunk-secret using below command:

      $ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>

      1. We have create clusterlogforwarders as below:

         ---

           apiVersion: "logging.openshift.io/v1"

           kind: "ClusterLogForwarder"

           metadata:

             name: "instance"

             namespace: "openshift-logging"

           spec:

             outputs:

               - name: splunk-receiver

                 secret:

                   name: vector-splunk-secret

                 type: splunk

                 url: http://splunk-hec.amosirelanddev.amosonline.io:8000

             pipelines:

               - inputRefs:

                   - application

                   - infrastructure

                 name:

                 outputRefs:

                   - splunk-receiver

         1. Updated cluster logging operator as it was using fluentd so replaced fluentd with vector:

            $ oc edit ClusterLogging instance -n openshift-logging

            Splunk Setup changes:

            1. Splunk installation on VM done with below steps:
               1. wget https://download.splunk.com/products/splunk/releases/8.0.4/linux/splunk-8.0.4-767223ac207f-linux-2.6...
               2. sudo rpm -ivh splunk-8.0.4-767223ac207f-linux-2.6-x86_64.rpm
               3. Two indexes created. Create new index as per below list. Settings > Indexes > New Index
                  1. openshift (events)
                  2. openshift-matrix (matix)
                     1. Enabling HEC token - Enable HEC (HTTP Event Collector), Settings > Data Inputs > HTTP Event Collector > Global Settings > Default Index as “Default” > Save
                     2. Create HEC token - Create new HEC token, Settings > Data inputs > HTTP Event Collector > New Token > Name as “openshift” > Next (Input Settings, add allowed indexes like below” > Review > Submit. Note the Token Value, we going to use this for next step.

                        We are trying to search from New Search “index= openshift” but not getting any result.

                        
                        

                          

                        Where we can see the logs on Splunk dashboard or if we are missing something then please let us know.

                        Regards,

                        Suchita Deshmukh

[Gr0und_Z3r0]

you can follow this document from splunk about securing your infrastructure.
https://docs.splunk.com/Documentation/Splunk/9.0.4/Security/AboutsecuringyourSplunkconfigurationwith... 
https://docs.splunk.com/Documentation/Splunk/9.0.4/Security/RenewExistingCerts 

If the reply helps, karma vote would be appreciated!

[Gr0und_Z3r0]

Check the port configured for HEC, it should be port 8088. You are hitting the web console on port 8000.

Some default ports..

8000 Web (default for clients to the Splunk Search page)
8089 Management/Rest API & Distributed Search (default)
9997 Indexing Receiver( for forwarders to the Splunk indexer)
8181 Search replication
8080 Index replication
8191 KV store/replication
8088 http Event Collector
8065 Splunk App Server
514 Legacy syslog input(UDP/TCP)
1433 DB Connector(to fetch data from databases to Splunk)

[Suchita1]

$ sudo netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 991/sshd
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 0.0.0.0:8191 0.0.0.0:* LISTEN 114889/mongod
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 127.0.0.1:8065 0.0.0.0:* LISTEN 114963/python3.7
tcp6 0 0 :::22 :::* LISTEN 991/sshd

I have updated port from 8000 to 8088

$ curl -v https://splunk-hec.amosirelanddev.amosonline.io:8088/services/collector
* Trying 176.34.143.107...
* TCP_NODELAY set
* Connected to splunk-hec.amosirelanddev.amosonline.io (176.34.143.107) port 8088 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It seams certs are not valid. Could you send me procedure how to update certs for splunk.

[Gr0und_Z3r0]

There could be many reasons for events not reaching Splunk. From network configurations to permissions from the cluster side or even port configuration for HEC. Have you tried sending a simple curl message using HEC token to the Splunk  from the cluster instance to see if its reaching?

[Suchita1]

Please provide steps to apply ssl certs for splunk..

[Gr0und_Z3r0]

hi @Suchita1 

You can refer the below documentation for SSL certs, based on your system architecture configure them accordingly.

https://docs.splunk.com/Documentation/Splunk/9.0.4/Security/AboutsecuringyourSplunkconfigurationwith...

Note: Select the documentation based on the Splunk version you are using. You can see it in the top right corner of product and version.

~ If the reply helps, a karma upvote would be appreciated.

[Suchita1]

Yes, I tried to execute below curl command

curl http://splunk-hec.amosirelanddev.amosonline.io:8000/en-GB/services/collector/event -d '{"event": "hello world"}'

curl http://splunk-hec.amosirelanddev.amosonline.io:8000/services/collector/event -d '{"event": "hello world"}'

but getting 303 Page not found! Error.

[GuruSairam]

Hi Team,

We installed the splunk with version 8.0.4 from scratch and created the clusterlogging and clusterlogforwarder instance with vector pointing to splunk vm.

Still we are unable to see the logs in the dashboard even sample logs are also not visible in the dashboard.

Regards,

Guru Sairam