Splunk Enterprise

How to troubleshoot why HTTP Event Collector is not listening?

bento_prod
Explorer

I am trying use the dockers Splunk logging driver.

I created an HTTP Event Collector and token. I tried to test it using telnet, but I am not getting a response.

curl -k https://<INSTANCE>:8088/services/collector -H 'Authorization: Splunk <TOKEN>' -d '{"event":"Hello, World!"}'

but I am not getting a response.

I also tried to run the the docker container like this:

docker run <DOCKERNAME> -p 8080:5000 --name=<DOCKERNAME> -h XXX -d \
--log-driver=splunk --log-opt splunk-token=<HTTP_COLLECTOR_TOKEN> --log-opt splunk-url=https://<INSTANCE>:8088

but could not see the logs being sent.

So,
a) I don't see that the Splunk instance is listening
b) I don't see the logs.

Any idea what I am missing?

0 Karma

gblock_splunk
Splunk Employee
Splunk Employee

Hi @bento_prod, check this link. For self-service cloud you need to prefix "input-"

0 Karma

gblock_splunk
Splunk Employee
Splunk Employee

@bento_prod, also in terms of Docker, the driver won't work today with self-service due to our certs not having support in golang. It will work with managed cloud instances though. If you want to use the driver with self-service, you'll need to deploy a forwarder running HEC (like in AWS) and have it forward to the cloud instance. We know this is not an ideal experience for the driver with single instance, and are working on a better solution.

0 Karma

particlebrandon
Explorer

@glbock Is this only an limit of Splunk Light? Wondering if this is also an limitation of Splunk Cloud also?

Having an the same problem with the Splunk Cloud Self-Service. Although I am currently in the free-trail period, so I am not sure if there is any limitations during this period.

0 Karma

gblock_splunk
Splunk Employee
Splunk Employee

HEC is available on self-service / trial. Can you show me your curl (obfuscate the hostname) i.e. input-xxxxx:8088/services/...

0 Karma

particlebrandon
Explorer

That is part of my confusion here, is the curl command works successfully for me. But when I try to use the docker log-driver I am getting an handshake failure. See the command below for an example of the docker run command I am using here.

Also when I tried to use the full URI including /services/collector for the splunk-url parameter it failed with an invalid format.

docker run --log-driver=splunk --log-opt splunk-token=C041DEEB-XXXX-XXX-9F5F-3XXXXXXXXXD1C --log-opt splunk-url=https://input-prd-p-5XXXXXXXXX.cloud.splunk.com:8088 --log-opt splunk-insecureskipverify=true hello-world
 docker: Error response from daemon: Failed to initialize logging driver: remote error: handshake failure.

I assed this same question via an forum post below. Just trying to get confirmation on what the correct course for using splunk cloud with the docker log-driver is.

Link to Splunk Forum Posting

0 Karma

Ahmed67
Engager

looks like you didn't define output groups.You must update your outputs.conf if need be so logs can be sent to event collector

0 Karma

bento_prod
Explorer

@ahmed67 I am using splunk cloud light. I dont have access to the outputs.conf

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!