topic Re: OpenShift Log Forwarding to Splunk in Dashboards & Visualizations

OpenShift Log Forwarding to Splunk?

Suchita1 — Tue, 28 Mar 2023 14:13:02 GMT

We are using OpenShift 4.11.27 and now looking for OpenShift Log Forwarding to Splunk.

Did below changes at OpenShift end to configure splunk:

Installed cluster-logging and elasticsearch-operator into OpenShift.
$ oc get csv -n openshift-logging

NAME                                     DISPLAY                            VERSION   REPLACES                                       PHASE

cluster-logging.v5.6.3                   Red Hat OpenShift Logging          5.6.3     cluster-logging.v5.6.2                         Succeeded

elasticsearch-operator.v5.6.3            OpenShift Elasticsearch Operator   5.6.3     elasticsearch-operator.v5.6.2                  Succeeded
1. Created secret vector-splunk-secret using below command:
  $ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>
  1. We have create clusterlogforwarders as below:
    ---
    
    apiVersion: "logging.openshift.io/v1"
    
    kind: "ClusterLogForwarder"
    
    metadata:
    
        name: "instance"
    
        namespace: "openshift-logging"
    
    spec:
    
        outputs:
    
          - name: splunk-receiver
    
            secret:
    
              name: vector-splunk-secret
    
            type: splunk
    
            url: http://splunk-hec.amosirelanddev.amosonline.io:8000
    
        pipelines:
    
          - inputRefs:
    
              - application
    
              - infrastructure
    
            name:
    
            outputRefs:
    
              - splunk-receiver
    1. Updated cluster logging operator as it was using fluentd so replaced fluentd with vector:
      $ oc edit ClusterLogging instance -n openshift-logging
      
      Splunk Setup changes:
      1. Splunk installation on VM done with below steps:
        
        wget https://download.splunk.com/products/splunk/releases/8.0.4/linux/splunk-8.0.4-767223ac207f-linux-2.6-x86_64.rpm
        
        sudo rpm -ivh splunk-8.0.4-767223ac207f-linux-2.6-x86_64.rpm
        
        Two indexes created. Create new index as per below list. Settings > Indexes > New Index
        
        openshift (events)
        
        openshift-matrix (matix)
        
        Enabling HEC token - Enable HEC (HTTP Event Collector), Settings > Data Inputs > HTTP Event Collector > Global Settings > Default Index as “Default” > Save
        
        Create HEC token - Create new HEC token, Settings > Data inputs > HTTP Event Collector > New Token > Name as “openshift” > Next (Input Settings, add allowed indexes like below” > Review > Submit. Note the Token Value, we going to use this for next step.
        
        We are trying to search from New Search “index= openshift” but not getting any result.
        
        Where we can see the logs on Splunk dashboard or if we are missing something then please let us know.
        
        Regards,
        
        Suchita Deshmukh

Re: OpenShift Log Forwarding to Splunk

Gr0und_Z3r0 — Thu, 23 Mar 2023 12:06:33 GMT

There could be many reasons for events not reaching Splunk. From network configurations to permissions from the cluster side or even port configuration for HEC. Have you tried sending a simple curl message using HEC token to the Splunk from the cluster instance to see if its reaching?

Re: OpenShift Log Forwarding to Splunk

Suchita1 — Thu, 23 Mar 2023 12:14:55 GMT

Yes, I tried to execute below curl command

curl http://splunk-hec.amosirelanddev.amosonline.io:8000/en-GB/services/collector/event -d '{"event": "hello world"}'

curl http://splunk-hec.amosirelanddev.amosonline.io:8000/services/collector/event -d '{"event": "hello world"}'

but getting 303 Page not found! Error.

Re: OpenShift Log Forwarding to Splunk

Gr0und_Z3r0 — Thu, 23 Mar 2023 12:58:32 GMT

Check the port configured for HEC, it should be port 8088. You are hitting the web console on port 8000.

Some default ports..

8000 Web (default for clients to the Splunk Search page)
8089 Management/Rest API & Distributed Search (default)
9997 Indexing Receiver( for forwarders to the Splunk indexer)
8181 Search replication
8080 Index replication
8191 KV store/replication
8088 http Event Collector
8065 Splunk App Server
514 Legacy syslog input(UDP/TCP)
1433 DB Connector(to fetch data from databases to Splunk)

Re: OpenShift Log Forwarding to Splunk

Suchita1 — Fri, 24 Mar 2023 11:52:08 GMT

$ sudo netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 991/sshd
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 0.0.0.0:8191 0.0.0.0:* LISTEN 114889/mongod
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 127.0.0.1:8065 0.0.0.0:* LISTEN 114963/python3.7
tcp6 0 0 :::22 :::* LISTEN 991/sshd

I have updated port from 8000 to 8088

$ curl -v https://splunk-hec.amosirelanddev.amosonline.io:8088/services/collector
* Trying 176.34.143.107...
* TCP_NODELAY set
* Connected to splunk-hec.amosirelanddev.amosonline.io (176.34.143.107) port 8088 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It seams certs are not valid. Could you send me procedure how to update certs for splunk.

Re: OpenShift Log Forwarding to Splunk

Gr0und_Z3r0 — Tue, 28 Mar 2023 09:00:09 GMT

you can follow this document from splunk about securing your infrastructure.
https://docs.splunk.com/Documentation/Splunk/9.0.4/Security/AboutsecuringyourSplunkconfigurationwithSSL
https://docs.splunk.com/Documentation/Splunk/9.0.4/Security/RenewExistingCerts

If the reply helps, karma vote would be appreciated!

Re: OpenShift Log Forwarding to Splunk

Suchita1 — Wed, 05 Apr 2023 05:01:13 GMT

Please provide steps to apply ssl certs for splunk..

Re: OpenShift Log Forwarding to Splunk

Gr0und_Z3r0 — Wed, 05 Apr 2023 08:03:55 GMT

hi @Suchita1

You can refer the below documentation for SSL certs, based on your system architecture configure them accordingly.

https://docs.splunk.com/Documentation/Splunk/9.0.4/Security/AboutsecuringyourSplunkconfigurationwithSSL

Note: Select the documentation based on the Splunk version you are using. You can see it in the top right corner of product and version.

~ If the reply helps, a karma upvote would be appreciated.

Re: OpenShift Log Forwarding to Splunk

GuruSairam — Mon, 16 Oct 2023 04:27:03 GMT

Hi Team,

We installed the splunk with version 8.0.4 from scratch and created the clusterlogging and clusterlogforwarder instance with vector pointing to splunk vm.

Still we are unable to see the logs in the dashboard even sample logs are also not visible in the dashboard.

Regards,

Guru Sairam