topic Re: enabling HTTP Event Collector for docker container on splunk clustered setup in Monitoring Splunk

enabling HTTP Event Collector for docker container on splunk clustered setup

sim_tcr — Mon, 20 Nov 2017 14:10:24 GMT

We tested HTTP Event Collector for a docker container by starting the container with below,

--log-driver=splunk --log-opt splunk-token=<token> --log-opt splunk-url=https://<splunkserver>:8088 --log-opt splunk-insecureskipverify=true

Now we want to send the events to our actual splunk setup. We are on search head clustering and index clustering enabled, with a separate shc deployer, deployment server and index master servers.

Which server we should enable the HTTP Event Collector?
How can i specify the index where the events should be forwarded?

Thanks,

Re: enabling HTTP Event Collector for docker container on splunk clustered setup

mattymo — Mon, 20 Nov 2017 15:00:42 GMT

Hi @sim_tcr!

The best advice for you would require a peek at your existing architecture and data volumes, but you should start by having a look at dev.splunk.com for HEC architectures:

http://dev.splunk.com/view/event-collector/SP-CAAAE73

If you are just getting started you could look at having a Heavy forwarder in place to catch the HEC traffic and pass it to your indexers, or a pool of HFs, or you could also just enable it on all your indexers - although you will want to be careful with that depending on the volume of traffic you expect.

As for routing to an index, check out the docs on HEC tokens. You can set the index/sourcetype there!

http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector

Re: enabling HTTP Event Collector for docker container on splunk clustered setup

sim_tcr — Mon, 20 Nov 2017 15:07:46 GMT

Thank you for responding.
I think we want to go with http://dev.splunk.com/view/event-collector/SP-CAAAE73#scen1
We are at 6.3.3 and referring http://docs.splunk.com/Documentation/Splunk/6.3.3/Data/UsetheHTTPEventCollector
when checking "select allowed indexes" field, we do not see all our indexes from the index cluster listed there.
Do you know what are we missing?

Re: enabling HTTP Event Collector for docker container on splunk clustered setup

mattymo — Mon, 20 Nov 2017 15:18:31 GMT

You need to tell your HF about them in indexes.conf

Grab the TA from the indexer cluster and use that indexes.conf on the HF.

Re: enabling HTTP Event Collector for docker container on splunk clustered setup

sloshburch — Wed, 22 Nov 2017 13:51:56 GMT

Great answer @mmodestino!

@sim_tcr - Remember that if you start with Scenario 1 and you have to change/rebuilt the HEC server, you'll need to update all the clients (apps) sending data. As such, use a VIP or a Load Balancer even if pointing to that one instance. This will increase availability and failover options besides allowing for scalability later and essentially making the transition to Scenario 3 seamless.

Re: enabling HTTP Event Collector for docker container on splunk clustered setup

outcoldman — Mon, 27 Nov 2017 06:01:52 GMT

Hi @sim_tcr,

If you are just integrating Docker with Splunk, have you seen our solution for collecting logs and metrics https://splunkbase.splunk.com/app/3723/? You can find some documentation https://www.outcoldsolutions.com/docs/ demos and screencasts. We have also comparison table with official Splunk Logging Driver https://www.outcoldsolutions.com/docs/collectorfordocker/#comparing-with-splunk-logging-driver (btw, I am the original author of this driver).