Is enabling HTTP Event Collector in a cluster possible via master apps files only instead of the web interface?

[wsanderstii]

I am trying to set up HEC in a cluster, but the cluster members do not seem to be listening on the port I have designated, 8088 via the web interface.
$ nmap -p 8088 splunk01.s2prod

Starting Nmap 6.49BETA5 ( https://nmap.org ) at 2018-02-12 10:00 PST
Nmap scan report for splunk01.s2prod (10.2.44.111)
Host is up (0.0046s latency).
PORT     STATE  SERVICE
8088/tcp closed radan-http

The documentation does not mention it, but is this another service that must be set up via the files in master-apps, rather than the web interface? We are still running 6.6.5.

Thanks, w

[outcoldman]

@wsanderstii I believe you should be able to configure Distributed Deployment in the way, that you will use Web UI on Deployment Server to configure HTTP Tokens and publish them to Indexing peer. But it can be tricky.

It is not recommended and not supported to use Indexer Cluster Peers as Deployment Clients of Deployment Server (you have seen this note in the documentation you have found).

But! You can use master cluster as a deployment client for deployment server. See http://docs.splunk.com/Documentation/Splunk/7.0.2/Updating/Aboutdeploymentserver

Indexer clusters

Do not use deployment server or forwarder management to manage configuration files across peer nodes (indexers) in an indexer cluster. Instead, use the configuration bundle method. You can, however, use the deployment server to distribute updates to the master node, which then uses the configuration bundle method to distribute them to the peer nodes. See "Update common peer configurations" in the Managing Indexers and Clusters of Indexers manual.

Look on the https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Deploymentclientconf with repositoryLocation and https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Serverclassconf with targetRepositoryLocation, so I assume you can give it a try and use master-apps as a repository location.

This is my assumption based on the documentation, and to be honest I am not sure if that actually works as documented, but you can give it a try.

[mattymo]

Yes, like all other configurations for clustered indexers, you need to push an app to the indexers that contains an inputs.conf with a HEC token, not via the GUI. http://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Clusterconfigurationoverview

This token should be identical across the indexers for each input you want to make.

Easiest way is to generate a HEC token in the GUI on a test machine with all the settings you want, then use btool ./splunk btool inputs list "<your_http_input>" --debug to see the running config and note how the GUID of the token looks.

For example:

[splunker@n00bserver bin]$ ./splunk btool inputs list "http://collectd" --debug
/home/splunker/splunk/etc/apps/k8s_backup/local/inputs.conf [http://collectd]
/home/splunker/splunk/etc/system/default/inputs.conf        _rcvbuf = 1572864
/home/splunker/splunk/etc/apps/k8s_backup/local/inputs.conf disabled = 0
/home/splunker/splunk/etc/system/local/inputs.conf          host = n00bserver.n00blab.local
/home/splunker/splunk/etc/apps/k8s_backup/local/inputs.conf index = em_metrics
/home/splunker/splunk/etc/apps/k8s_backup/local/inputs.conf indexes = em_metrics
/home/splunker/splunk/etc/apps/k8s_backup/local/inputs.conf sourcetype = collectd_http
/home/splunker/splunk/etc/apps/k8s_backup/local/inputs.conf token = 00000000-0000-0000-0000-000000000000

As you can see, the configs can end up all over the place. So as we say around here, btool is your friend!

to see all your tokens use ./splunk btool inputs list http --debug

This will show you some common settings that are available, but also check the spec file for more detailed info on all your options : http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf#http:_.28HTTP_Event_Collector.29

You can create any 32bit GUID you want in the pattern above (dont use all 0s, but that will definitely work!) and distribute that same token to each of the indexers in an app. Be aware that if you have a GUI on the indexers (you probably shouldnt), community testing shows it won't reflect the token as enabled unless it's in the splunk_httpinput app. You should disgregard this anyways as clusterded indexers shouldn't be really viewed from the GUI anyways.

You can test your config has been deployed successfully by sending a test curl command:

curl -k https://:8088/services/collector -H 'Authorization: Splunk ' -d '{"sourcetype": "mysourcetype", "event":"Hello, World!"}'


http://dev.splunk.com/view/event-collector/SP-CAAAE7F

One useful note on SSL. HEC uses the same certs as splunkd in server.conf by default. If you are going to use SSL with your own certs, you can define them in inputs.conf, or you can define the global server.conf settings to use 3rd party certs, which HEC will use by default with ssl is enabled. The "right" decision depends on your architecture and environment, but most deploys leave splunkd's certs alone for inter splunk communication, and just define them in the inputs conf. My lab is standalone, so I just replaced splunkd's cert in server.conf. See the settings in the inputs.conf.spec file I posted above.

[wsanderstii]

I am looking at this document: http://dev.splunk.com/view/event-collector/SP-CAAAE73, which states "Note: Using HTTP Event Collector in a distributed deployment is incompatible with indexer clustering. Specifically, cluster peers are not supported as deployment clients."
I am not really sure what that means; but it seems like Scenario 2 ought to work if we are willing to send data asymmetrically to only one indexer. And it seems like we would not need a deployment server if the indexers were clustered and the configs were pushed out from master-apps/.
Or is this document obsolete?

[wsanderstii]

Somehow, this was made to work by my predecessor who sort of half set up the cluster a few years ago:

• Enabled HEC on the cluster master; port 8088 is open on the cluster master.
• JSON is sent to the cluster master on port 8088:curl -k https://splunk-master:8088 -H 'Authorization: Splunk '<token>' -d '{"sourcetype": "mysourcetype", "event":"Hello, World!"}'
• Events show up in indexes on the cluster master and also magically (*) on one or more of the clustered indexer nodes.

(*) I think it's possible the master is configured as a forwarder.