We are using OpenShift 4.11.27 and now looking for OpenShift Log Forwarding to Splunk.
Did below changes at OpenShift end to configure splunk:
$ oc get csv -n openshift-logging
NAME DISPLAY VERSION REPLACES PHASE
cluster-logging.v5.6.3 Red Hat OpenShift Logging 5.6.3 cluster-logging.v5.6.2 Succeeded
elasticsearch-operator.v5.6.3 OpenShift Elasticsearch Operator 5.6.3 elasticsearch-operator.v5.6.2 Succeeded
$ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>
---
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogForwarder"
metadata:
name: "instance"
namespace: "openshift-logging"
spec:
outputs:
- name: splunk-receiver
secret:
name: vector-splunk-secret
type: splunk
url: http://splunk-hec.amosirelanddev.amosonline.io:8000
pipelines:
- inputRefs:
- application
- infrastructure
name:
outputRefs:
- splunk-receiver
$ oc edit ClusterLogging instance -n openshift-logging
Splunk Setup changes:
We are trying to search from New Search “index= openshift” but not getting any result.
Where we can see the logs on Splunk dashboard or if we are missing something then please let us know.
Regards,
Suchita Deshmukh
you can follow this document from splunk about securing your infrastructure.
https://docs.splunk.com/Documentation/Splunk/9.0.4/Security/AboutsecuringyourSplunkconfigurationwith...
https://docs.splunk.com/Documentation/Splunk/9.0.4/Security/RenewExistingCerts
If the reply helps, karma vote would be appreciated!
Check the port configured for HEC, it should be port 8088. You are hitting the web console on port 8000.
Some default ports..
8000 Web (default for clients to the Splunk Search page)
8089 Management/Rest API & Distributed Search (default)
9997 Indexing Receiver( for forwarders to the Splunk indexer)
8181 Search replication
8080 Index replication
8191 KV store/replication
8088 http Event Collector
8065 Splunk App Server
514 Legacy syslog input(UDP/TCP)
1433 DB Connector(to fetch data from databases to Splunk)
$ sudo netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 991/sshd
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 0.0.0.0:8191 0.0.0.0:* LISTEN 114889/mongod
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 114868/splunkd
tcp 0 0 127.0.0.1:8065 0.0.0.0:* LISTEN 114963/python3.7
tcp6 0 0 :::22 :::* LISTEN 991/sshd
I have updated port from 8000 to 8088
$ curl -v https://splunk-hec.amosirelanddev.amosonline.io:8088/services/collector
* Trying 176.34.143.107...
* TCP_NODELAY set
* Connected to splunk-hec.amosirelanddev.amosonline.io (176.34.143.107) port 8088 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
It seams certs are not valid. Could you send me procedure how to update certs for splunk.
There could be many reasons for events not reaching Splunk. From network configurations to permissions from the cluster side or even port configuration for HEC. Have you tried sending a simple curl message using HEC token to the Splunk from the cluster instance to see if its reaching?
Please provide steps to apply ssl certs for splunk..
hi @Suchita1
You can refer the below documentation for SSL certs, based on your system architecture configure them accordingly.
https://docs.splunk.com/Documentation/Splunk/9.0.4/Security/AboutsecuringyourSplunkconfigurationwith...
Note: Select the documentation based on the Splunk version you are using. You can see it in the top right corner of product and version.
~ If the reply helps, a karma upvote would be appreciated.
Yes, I tried to execute below curl command
curl http://splunk-hec.amosirelanddev.amosonline.io:8000/en-GB/services/collector/event -d '{"event": "hello world"}'
curl http://splunk-hec.amosirelanddev.amosonline.io:8000/services/collector/event -d '{"event": "hello world"}'
but getting 303 Page not found! Error.
Hi Team,
We installed the splunk with version 8.0.4 from scratch and created the clusterlogging and clusterlogforwarder instance with vector pointing to splunk vm.
Still we are unable to see the logs in the dashboard even sample logs are also not visible in the dashboard.
Regards,
Guru Sairam