topic Re: Securing HTTP Event Collector in All Apps and Add-ons

How do I secure the event collector port 8088 with an ssl certificate?

brent_weaver — Tue, 03 May 2022 20:16:11 GMT

I need to use the HTTP Even Collector for ingestion and it will be exposed to the web so I need to secure this communications. How do I secure the event collector port 8088 with an ssl certificate?

Re: Securing HTTP Event Collector

hunters_splunk — Wed, 12 Oct 2016 13:59:29 GMT

Hi Brent_weaver,

Please refer to this topic:

http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/UsetheHTTPEventCollector

Step 9: To have HEC listen and communicate over HTTPS rather than HTTP, click the Enable SSL checkbox.

Hope it helps.
Thanks!
Hunter Shen

Re: Securing HTTP Event Collector

mshenoyp — Tue, 29 Sep 2020 11:26:28 GMT

Hi @hunters,

Can you share an example as to where to configure the ssl certificate as with documentation its not clear.

BTW i tried to set the sslconfig under $SPLUNK_HOME/etc/apps/splunk_httpinput/local/server.conf but in vain :(. Also tried the sslconfig under $SPLUNK_HOME/etc/system/local/server.conf but no luck :(.

server.conf details

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
sslKeysfile = myServerCert.pem
caCertFile = myCacert.pem
caPath = $SPLUNK_HOME/etc/auth/actualCerts

Re: Securing HTTP Event Collector

kngklla — Wed, 18 Jan 2017 11:36:30 GMT

Bump for an answer. http://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Inputsconf#http:_.28HTTP_Event_Collector.29 may be useful, but an article dedicated to the nuances of the SSL cert configuration for HEC would be much appreciated.

Re: Securing HTTP Event Collector

amiracle — Fri, 11 Aug 2017 02:50:31 GMT

The HTTP Event Collector uses the SSL Cert settings from the [general] stanza of the $SPLUNK_HOME/etc/system/local/server.conf file. See the docs : HEC Docs

enableSSL

Whether the HTTP Event
Collector server protocol is HTTP or
HTTPS. 1 indicates HTTPS; 0 indicates
HTTP. The default value is 1. If you
set enableSSL to 1, HTTP Event
Collector uses the SSL settings
specified in the [general] stanza of
the server.conf file.

Re: Securing HTTP Event Collector

chinmayad — Tue, 22 Aug 2017 22:19:25 GMT

Was anyone here able to get the HEC working with SSL?
The above documentation points to "the [general] stanza of the $SPLUNK_HOME/etc/system/local/server.conf file".
However, the general stanza does not include options for feeding the path to the server.crt or the private server.key.
Getting the ssl to work for the web server was much easier and well documented. I just had to do

[settings]
enableSplunkWebSSL = true
privKeyPath = $SPLUNK_HOME/etc/auth/mycerts/server.key
serverCert = $SPLUNK_HOME/etc/auth/mycerts/server.crt

in web.conf and load the certificate and the key to the above location.

I am using third party certs btw. Can someone point me to a way to feed the certificate and the private key to the HEC?

Re: Securing HTTP Event Collector

melcher — Tue, 22 Aug 2017 22:38:42 GMT

I ran into this issue a bit myself. What I found needed to happen was to configure these settings in $SPLUNK_HOME/etc/system/local/server.conf

[sslConfig]
caCertFile = <ABSOLUTE_PATH_TO_CA_FILE>
serverCert = <ABSOLUTE_PATH_TO_SERVER_CERT_FILE>
sslPassword = <PASSWORD>

The key that I had to take note of was that the "sslPassword" setting needed to be present. Otherwise it inherited the default "sslPassword" setting and failed to initialize SSL. If your third party SSL cert didn't come with a password, you can create one using openssl like this:

openssl rsa -in splunk_ssl.pem -out splunk_ssl.pem -des3

Re: Securing HTTP Event Collector

melcher — Wed, 23 Aug 2017 15:28:30 GMT

Check my above answer to see if that helps you out.

Re: Securing HTTP Event Collector

cxj — Tue, 12 Sep 2017 13:35:41 GMT

This is the exact same process we followed - just remember to put your cert on each of the servers for the tier you're getting data in from; i.e. Heavy Forwarders or Indexing tier based upon which option you chose and make sure the certs are in the exact same path.

FYI we left our sslPassword option empty and once we restarted the splunk process it auto-magically populated the information with a password.

Re: Securing HTTP Event Collector

Meertman — Thu, 14 Sep 2017 15:38:01 GMT

I followed the exact same procedure.

I configured the SSL certificate in the server.conf of the forwarder using the [sslConfig] section, but the HTTP event collector is still presenting my clients with the Splunk generated certificate.

I have tried checking the error logs from Splunk to see if something was wrong and indeed the first few attempts, I found that my certificate was incorrect. (The following post helped me to figure this out: https://answers.splunk.com/answers/402988/after-upgrading-to-64-why-are-our-signed-certs-no.html)

However I do not receive any errors anymore in my logs, but still the forwarder is presenting my clients with the old Splunk generated certificate. Can someone tell me what I am doing wrong?

Thanks in advance!

Re: Securing HTTP Event Collector

cxj — Tue, 29 Sep 2020 15:45:15 GMT

Are your customers going through a VIP or going directly to the HF or Indexer? Not too sure how you're setup, but we're using a VIP in front of our HF tier. Once we applied the changes to EACH HF and the customer used the full FQDN of the VIP then they were able to get a valid cert.

The Splunk docs state to use a .PEM file, but we used a .CRT and .KEY file and that worked like a champ. Note: we're on Splunk 6.3.6 so your version/config may differ so check the splunk docs.

Below what our server.conf looks like before we restarted splunkd:
[sslConfig]
caCertFile = cert_name.crt
caPath = path_to_cert
sslKeysfile = cert_name.key
sslKeysfilePassword =

Lastly - make sure you update your inputs.conf for splunk_http to leverage SSL - note that we have a clustered server so this is configured on our deployment server and then pushed to our HF tier via serverclass.conf:

[http]
dedicatedIoThreads = 2
disabled = 0
enableSSL = 1
index = default
port = 8090
host = deployment_server_name
maxSockets = 0
maxThreads = 0

Re: Securing HTTP Event Collector

Meertman — Mon, 18 Sep 2017 07:47:52 GMT

Thanks for the input cxj. We are not going through a VIP as of yet and I am currently testing it with one universal forwarder in our DMZ.

I did use a PEM, that included the public key, the private key, all intermediate CA's public keys and the public key of the CA. (which resolved all certificate errors in the logs).

I did not provide the caPath or caCertFile as the documentation states that you are not required to do so on a Windows platform. (The version we are currently using is 6.4.1)

Re: Securing HTTP Event Collector

robwhite — Mon, 16 Jul 2018 06:55:17 GMT

melcher, i'm a little unclear on the files here.

I have a certificate, a chain (intermediate certs) and a private key. The private key is password protected.

How does this relate to the options caCertFile, serverCert and sslPassword? I see no option to specify the path to my private key.

Re: Securing HTTP Event Collector

melcher — Mon, 16 Jul 2018 15:52:04 GMT

You don't include the private key. Splunk doesn't use it. The cert itself needs to have a password applied to it. And then the cert and intermediate cert chains are what are used in the "caCertFile" and "serverCert" settings.

Re: Securing HTTP Event Collector

robwhite — Tue, 17 Jul 2018 03:36:01 GMT

This seems a little bizarre. I'm not sure how Splunk could ever not use a private key - otherwise what stops me just downloading the google.com cert and using that for my HEC?

I found this blog which does use the private key and works for me https://support.uberagent.com/hc/en-us/articles/212656725-Enable-HTTPS-for-Splunk-HTTP-Event-Collector

Re: Securing HTTP Event Collector

ischoenmaker — Thu, 18 Oct 2018 11:31:32 GMT

I've just spent some time with the same challenge and found out most of the above might not be the best solution, simply does not work, or is deprecated.

By default when you enable HEC and choose enable SSL Splunk uses the same self signed certs as for port 8089.

To secure HEC you can change this configurationin server.conf. Of course it might not be the best idea to secure your HEC with the same certs as port 8089 management traffic.

We solved it by generating a new certification chain (resulting in a new pem with server cert + key + CA cert) and key and edit the HEC stanza in inputs.conf:

[http]
disabled = 0
index = main
enableSSL = 1
serverCert = <full path to your certificate chain pem file>
sslPassword = <password for server key used in chain>

Restart splunkd and now your HEC uses this cert.

For reference see:
http://docs.splunk.com/Documentation/Splunk/7.2.0/Admin/Inputsconf#http:_.28HTTP_Event_Collector.29

As said Splunk expects a chain of certs and key, see the following examples for how to format this for Splunk and how to generate them self signed:
http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates
http://docs.splunk.com/Documentation/Splunk/7.2.0/Security/HowtoprepareyoursignedcertificatesforSplunk

Re: Securing HTTP Event Collector

bpaul_splunk — Tue, 22 Jan 2019 19:22:37 GMT

As of more recent versions of Splunk (version 7.x and newer I believe) this works.

Re: Securing HTTP Event Collector

jackjack — Tue, 19 Oct 2021 00:06:34 GMT

This answer was the most helpful for me. I am adding a few things I found helpful for anyone using Certbot/LetsEncrypt

1. Generate the pem key using the letsencrypt certs

cd /etc/letsencrypt/live/your-server-hostname/ cat cert.pem privkey.pem chain.pem > splunk.pem chmod 777 splunk.pem

2. Use the following for inputs.conf

[http] disabled = 0 index = your-hec-index-name enableSSL = 1 serverCert = /etc/letsencrypt/live/your-server-hostname/splunk.pem sslPassword = crossOriginSharingPolicy = *

3. Troubleshoot the connection

This comes from this forum post

curl -k https://your-heavy-forwarder.com:8088/services/collector/event -H "Authorization: Splunk splunk-hec-token" -d '{"event":"hello world"}' -v

Re: Securing HTTP Event Collector

dstromberg — Tue, 03 May 2022 19:52:04 GMT

Which inputs.conf are you talking about?

I have:

[root@splunk splunk]# find /opt/splunk/etc/ -name inputs.conf -print
/opt/splunk/etc/modules/distributedDeployment/classes/deployable/inputs.conf
/opt/splunk/etc/apps/splunk_secure_gateway/default/inputs.conf
/opt/splunk/etc/apps/splunk-dashboard-studio/default/inputs.conf
/opt/splunk/etc/apps/SplunkLightForwarder/default/inputs.conf
/opt/splunk/etc/apps/sample_app/default/inputs.conf
/opt/splunk/etc/apps/journald_input/default/inputs.conf
/opt/splunk/etc/apps/introspection_generator_addon/default/inputs.conf
/opt/splunk/etc/apps/splunk_instrumentation/default/inputs.conf
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf
/opt/splunk/etc/apps/splunk_monitoring_console/default/inputs.conf
/opt/splunk/etc/apps/python_upgrade_readiness_app/default/inputs.conf
/opt/splunk/etc/system/local/inputs.conf
/opt/splunk/etc/system/default/inputs.conf

Thanks.

Re: Securing HTTP Event Collector

bullet — Sat, 28 May 2022 04:19:17 GMT

@dstromberg

See this link

it is under this directory, if you don't see input.conf then I guess you can create it there.

cd /opt/splunk/etc/system/local/