Configuring Splunk and Rancher

[emcclure]

Hello,

I'm totally new to this and have been thrown into the fire to figure things out. I'm setting up Kubernetes with Rancher and want to integrate splunk monitoring with it. That's where I'm stuck. I've followed the steps here: https://rancher.com/docs/rancher/v2.x/en/tools/logging/splunk/ and also looked at these docs for creating the event collector tokens in splunk: https://docs.splunk.com/Documentation/Splunk/7.0.0/Data/UsetheHTTPEventCollector#Configure_HTTP_Even..., but I'm not seeing anything at all in splunk. I'm not sure if I've misconfigured something or what the deal is, but I've been looking around on the Rancher forums and on their slack channel and have received no help at all on it, so I'm hoping that someone can help me out here.

I've never setup splunk before for Kubernetes and this is a totally new project, trial by fire you might say. Any help is appreciated. Please let me know if any other info is needed.

[rudyj03]

Hi,

I'm running into the same issue, were you able to resolve it? In my case I have rancher set up properly and clicking Test from the logging setup page results in a sample message in Splunk. However, my container logs themselves don't show up in splunk at all. I've made sure I can run the above curl command from the machine that the rancher server is running on as well.

[ifeldshteyn]

Hey,

I've setup Splunk logging with Rancher. First you want to make sue HEC is running correctly. Using that HEC guide you linked, make sure you can curl it. If the below doesn't work from the server hosting Rancher nodes then you have a firewall issue.

curl -k https://hec.example.com:8088/services/collector/event -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
{"text": "Success", "code": 0}

Then choose a cluster that spits out some output and set up your endpoint and server. It takes 30 sec usually for it to start sending data. If you are not seeing any, it's possible you may have a firewall. Make sure 8088 is available.

alt text

[ifeldshteyn]

Yeah Port 8000 is used for the web interface only. You should use port 8088 for your HEC.

Splunk by default uses a bunch of ports, 8000 for gui, 8089 for management, 7999 for forwarding, 8088 for HEC.

[emcclure]

Ok so if I understand this right in the command the https://hec.example.com is the name of the splunk server? And the port is what's used when logging in thru the browser, which I have at 8000, or should it be 8088? Then I need to specify the token name and token value as well where it says Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0? Here's what I get when I try this:

curl -k http://mysplunk.server:8000/services/collector/event -H "Authorization: Rancher " -d '{"event": "hello world"}' {"text": "Success", "code": 0}

I get this:

See Other

The resource has moved temporarily here.

curl: (3) [globbing] unmatched brace at pos 7
curl: (6) Could not resolve host: Success,; Unknown error
curl: (6) Could not resolve host: code; Unknown error
curl: (3) [globbing] unmatched close brace/bracket at pos 2

If I try it like this:
curl -k http://mysplunk.server:8088/services/collector/event -H "Authorization: Rancher " -d '{"event": "hello world"}' {"text": "Success", "code": 0}

I get this:
curl: (52) Empty reply from server
curl: (3) [globbing] unmatched brace at pos 7
curl: (6) Could not resolve host: Success,; Unknown error
curl: (6) Could not resolve host: code; Unknown error
curl: (3) [globbing] unmatched close brace/bracket at pos 2

If I make the command https and use this:
curl -k https://mysplunk.server:8000/services/collector/event -H "Authorization: Rancher " -d '{"event": "hello world"}' {"text": "Success", "code": 0}

I get this:
curl: (35) SSL received a record that exceeded the maximum permissible length.
curl: (3) [globbing] unmatched brace at pos 7
curl: (6) Could not resolve host: Success,; Unknown error
curl: (6) Could not resolve host: code; Unknown error
curl: (3) [globbing] unmatched close brace/bracket at pos 2

And finally if I use the same command but change the port to 8088 I get this:

{"text":"Invalid authorization","code":3}curl: (3) [globbing] unmatched brace at pos 7
curl: (6) Could not resolve host: Success,; Unknown error
curl: (6) Could not resolve host: code; Unknown error
curl: (3) [globbing] unmatched close brace/bracket at pos 2

So what am I doing wrong here?

[ifeldshteyn]

Hello,

1. Do not use port 8000, use 8088. You should only use 8000 for your web interface if you want to look at GUI stuff.
2. Your authorization should be some kind of a long hex token like A832455B-DE61-B4B1-B104-5FAA46315CF2 not Rancher
3. Don't append {"text": "Success", "code": 0} that's the output you should receive if you set up your HEC correctly.

Please look over instructions here, it gives you examples at the bottom
https://docs.splunk.com/Documentation/Splunk/7.0.0/Data/UsetheHTTPEventCollector#Configure_HTTP_Even...

It should be something like this, assuming you are running on https and not http.

curl -k https://hec.example.com:8088/services/collector/event -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'

The only thing you need to replace is the server name from hec.example.com to yourservername and the B5A... to your own token.

[rudyj03]

Hi,

I'm running into the same issue, were you able to resolve it? In my case I have rancher set up properly and clicking Test from the logging setup page results in a sample message in Splunk. However, my container logs themselves don't show up in splunk at all. I've made sure I can run the above curl command from the machine that the rancher server is running on as well.

[emcclure]

Ok I was able to build a new setup with Rancher and try and configure splunk again. However this time when I run the command I get this:

{"text":"Data channel is missing","code":10}

When I setup the HEC part in Rancher I used port 8088 for the endpoint and the token I got when I configured it. For the index I used the one I created in splunk and I left the Source field blank as I wasn't sure what needed to go there, and it saved anyway. What am I doing wrong? Is there something I'm missing still?

[emcclure]

Ok I'll do that, but I'll have to create a new setup of Rancher. Had to re-ip some hosts and it's caused nothing but issues for me. Rancher setup seems to have somehow 'reset' itself and wants me to basically start all over, as if I never did anything at all. So when I setup splunk in Rancher and I list the endpoint I should use port 8088? And then when I run the command as shown above just modify the hec.example.com to the splunk server and then the token I used to connect the splunk server to Rancher? If it's running http does that cause any issues? I'm guessing just changing the curl command to http is ok, or is more involved?

[emcclure]

Ok I will try that out. The port used in the web address is 8000. Is that an issue at all or do I need to specifiy port 8088 no matter what?