How do I secure the event collector port 8088 with an ssl certificate?

[brent_weaver]

I need to use the HTTP Even Collector for ingestion and it will be exposed to the web so I need to secure this communications. How do I secure the event collector port 8088 with an ssl certificate?

[ischoenmaker]

I've just spent some time with the same challenge and found out most of the above might not be the best solution, simply does not work, or is deprecated.

By default when you enable HEC and choose enable SSL Splunk uses the same self signed certs as for port 8089.

To secure HEC you can change this configurationin server.conf. Of course it might not be the best idea to secure your HEC with the same certs as port 8089 management traffic.

We solved it by generating a new certification chain (resulting in a new pem with server cert + key + CA cert) and key and edit the HEC stanza in inputs.conf:

[http]
disabled = 0
index = main
enableSSL = 1
serverCert = <full path to your certificate chain pem file>
sslPassword = <password for server key used in chain>

Restart splunkd and now your HEC uses this cert.

For reference see:
http://docs.splunk.com/Documentation/Splunk/7.2.0/Admin/Inputsconf#http:_.28HTTP_Event_Collector.29

As said Splunk expects a chain of certs and key, see the following examples for how to format this for Splunk and how to generate them self signed:
http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates
http://docs.splunk.com/Documentation/Splunk/7.2.0/Security/HowtoprepareyoursignedcertificatesforSplu...

[dstromberg]

Which inputs.conf are you talking about?

I have:

[root@splunk splunk]# find /opt/splunk/etc/ -name inputs.conf -print
/opt/splunk/etc/modules/distributedDeployment/classes/deployable/inputs.conf
/opt/splunk/etc/apps/splunk_secure_gateway/default/inputs.conf
/opt/splunk/etc/apps/splunk-dashboard-studio/default/inputs.conf
/opt/splunk/etc/apps/SplunkLightForwarder/default/inputs.conf
/opt/splunk/etc/apps/sample_app/default/inputs.conf
/opt/splunk/etc/apps/journald_input/default/inputs.conf
/opt/splunk/etc/apps/introspection_generator_addon/default/inputs.conf
/opt/splunk/etc/apps/splunk_instrumentation/default/inputs.conf
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf
/opt/splunk/etc/apps/splunk_monitoring_console/default/inputs.conf
/opt/splunk/etc/apps/python_upgrade_readiness_app/default/inputs.conf
/opt/splunk/etc/system/local/inputs.conf
/opt/splunk/etc/system/default/inputs.conf

Thanks.

[PickleRick]

This is a very old thread. You can get more visibiliy by starting a new one instead.

But to the point - all those files are relevant. Splunk builds a single configuration version from various files in its directories according to precedence rules - https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/Wheretofindtheconfigurationfiles

[bullet]

@dstromberg 

See this link

it is under this directory, if you don't see input.conf then I guess you can create it there.

cd /opt/splunk/etc/system/local/

[jackjack]

This answer was the most helpful for me. I am adding a few things I found helpful for anyone using Certbot/LetsEncrypt

1. Generate the pem key using the letsencrypt certs

cd /etc/letsencrypt/live/your-server-hostname/
cat cert.pem privkey.pem chain.pem > splunk.pem
chmod 777 splunk.pem

2. Use the following for inputs.conf

[http]
disabled = 0
index = your-hec-index-name
enableSSL = 1
serverCert = /etc/letsencrypt/live/your-server-hostname/splunk.pem
sslPassword = 
crossOriginSharingPolicy = *

3. Troubleshoot the connection

This comes from this forum post

curl -k https://your-heavy-forwarder.com:8088/services/collector/event -H "Authorization: Splunk splunk-hec-token" -d '{"event":"hello world"}' -v

[bpaul_splunk]

As of more recent versions of Splunk (version 7.x and newer I believe) this works.

[melcher]

I ran into this issue a bit myself. What I found needed to happen was to configure these settings in $SPLUNK_HOME/etc/system/local/server.conf

[sslConfig]
caCertFile = <ABSOLUTE_PATH_TO_CA_FILE>
serverCert = <ABSOLUTE_PATH_TO_SERVER_CERT_FILE>
sslPassword = <PASSWORD>

The key that I had to take note of was that the "sslPassword" setting needed to be present. Otherwise it inherited the default "sslPassword" setting and failed to initialize SSL. If your third party SSL cert didn't come with a password, you can create one using openssl like this:

openssl rsa -in splunk_ssl.pem -out splunk_ssl.pem -des3

[cxj]

This is the exact same process we followed - just remember to put your cert on each of the servers for the tier you're getting data in from; i.e. Heavy Forwarders or Indexing tier based upon which option you chose and make sure the certs are in the exact same path.

FYI we left our sslPassword option empty and once we restarted the splunk process it auto-magically populated the information with a password.

[robwhite]

melcher, i'm a little unclear on the files here.

I have a certificate, a chain (intermediate certs) and a private key. The private key is password protected.

How does this relate to the options caCertFile, serverCert and sslPassword? I see no option to specify the path to my private key.

[melcher]

You don't include the private key. Splunk doesn't use it. The cert itself needs to have a password applied to it. And then the cert and intermediate cert chains are what are used in the "caCertFile" and "serverCert" settings.

[robwhite]

This seems a little bizarre. I'm not sure how Splunk could ever not use a private key - otherwise what stops me just downloading the google.com cert and using that for my HEC?

I found this blog which does use the private key and works for me https://support.uberagent.com/hc/en-us/articles/212656725-Enable-HTTPS-for-Splunk-HTTP-Event-Collect...

[Meertman]

I followed the exact same procedure.

I configured the SSL certificate in the server.conf of the forwarder using the [sslConfig] section, but the HTTP event collector is still presenting my clients with the Splunk generated certificate.

I have tried checking the error logs from Splunk to see if something was wrong and indeed the first few attempts, I found that my certificate was incorrect. (The following post helped me to figure this out: https://answers.splunk.com/answers/402988/after-upgrading-to-64-why-are-our-signed-certs-no.html)

However I do not receive any errors anymore in my logs, but still the forwarder is presenting my clients with the old Splunk generated certificate. Can someone tell me what I am doing wrong?

Thanks in advance!

[cxj]

Are your customers going through a VIP or going directly to the HF or Indexer? Not too sure how you're setup, but we're using a VIP in front of our HF tier. Once we applied the changes to EACH HF and the customer used the full FQDN of the VIP then they were able to get a valid cert.

The Splunk docs state to use a .PEM file, but we used a .CRT and .KEY file and that worked like a champ. Note: we're on Splunk 6.3.6 so your version/config may differ so check the splunk docs.

Below what our server.conf looks like before we restarted splunkd:
[sslConfig]
caCertFile = cert_name.crt
caPath = path_to_cert
sslKeysfile = cert_name.key
sslKeysfilePassword =

Lastly - make sure you update your inputs.conf for splunk_http to leverage SSL - note that we have a clustered server so this is configured on our deployment server and then pushed to our HF tier via serverclass.conf:

[http]
dedicatedIoThreads = 2
disabled = 0
enableSSL = 1
index = default
port = 8090
host = deployment_server_name
maxSockets = 0
maxThreads = 0

[Meertman]

Thanks for the input cxj. We are not going through a VIP as of yet and I am currently testing it with one universal forwarder in our DMZ.

I did use a PEM, that included the public key, the private key, all intermediate CA's public keys and the public key of the CA. (which resolved all certificate errors in the logs).

I did not provide the caPath or caCertFile as the documentation states that you are not required to do so on a Windows platform. (The version we are currently using is 6.4.1)

[chinmayad]

Was anyone here able to get the HEC working with SSL?
The above documentation points to "the [general] stanza of the $SPLUNK_HOME/etc/system/local/server.conf file".
However, the general stanza does not include options for feeding the path to the server.crt or the private server.key.
Getting the ssl to work for the web server was much easier and well documented. I just had to do

[settings]
enableSplunkWebSSL = true
privKeyPath = $SPLUNK_HOME/etc/auth/mycerts/server.key
serverCert = $SPLUNK_HOME/etc/auth/mycerts/server.crt

in web.conf and load the certificate and the key to the above location.

I am using third party certs btw. Can someone point me to a way to feed the certificate and the private key to the HEC?

[melcher]

Check my above answer to see if that helps you out.

[amiracle]

The HTTP Event Collector uses the SSL Cert settings from the [general] stanza of the $SPLUNK_HOME/etc/system/local/server.conf file. See the docs : HEC Docs

enableSSL

Whether the HTTP Event
Collector server protocol is HTTP or
HTTPS. 1 indicates HTTPS; 0 indicates
HTTP. The default value is 1. If you
set enableSSL to 1, HTTP Event
Collector uses the SSL settings
specified in the [general] stanza of
the server.conf file.

[kngklla]

Bump for an answer. http://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Inputsconf#http:_.28HTTP_Event_Collector.29 may be useful, but an article dedicated to the nuances of the SSL cert configuration for HEC would be much appreciated.

[hunters_splunk]

Hi Brent_weaver,

Please refer to this topic:

http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/UsetheHTTPEventCollector

Step 9: To have HEC listen and communicate over HTTPS rather than HTTP, click the Enable SSL checkbox.

Hope it helps.
Thanks!
Hunter Shen

[mshenoyp]

Hi @hunters,

Can you share an example as to where to configure the ssl certificate as with documentation its not clear.

BTW i tried to set the sslconfig under $SPLUNK_HOME/etc/apps/splunk_httpinput/local/server.conf but in vain :(. Also tried the sslconfig under $SPLUNK_HOME/etc/system/local/server.conf but no luck :(.

server.conf details

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
sslKeysfile = myServerCert.pem
caCertFile = myCacert.pem
caPath = $SPLUNK_HOME/etc/auth/actualCerts