Problem ingesting from HEC, sslv3 alert certificate unknown

[manderson7]

We're attempting to ingest from ELK servers into Splunk using ELK -> HEC, but are having difficulties getting past ssl. Due to FW constraints, we're only able to send to one heavy forwarder on port 8088, which already has ssl enabled. We don't have certificate validation enabled.
The data is going to be coming in from a company that we've purchased, so they're not on our domain, and I'm not certain if their root ca cert is in effect in our domain, nor certain if that matters here.
They are sending using the following structure:

http_method => "put"
format => "json"
url => https://nattdip:8088/services/collector 
headers => {"Authorization" =>"Bearer d****d-9f84-4a3a-a9fd-6*******e"}
content_type => "application/json"

We've tried both put and post as the method, and they get the same error:

[HTTP Output Failure] Could not fetch URL {:url=>"https://nattdip:8088/services/collector", :method=>:post,

I see the following in my _internal log:

08-28-2018 15:45:13.287 -0400 WARN HttpListener - Socket error from sourceip while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown

We've added their rootca to our pem file that Splunk is using to protect web & hec, but get same error.

Any suggestions would be great. Thanks very much.

[harsmarvania57]

Hi @manderson7,

If you are using HTTP Output Plugin on Logstash then you can configure cacert parameter on logstash server, for more info see this https://discuss.elastic.co/t/logstash-to-splunk-http-event-collector/130765/3

[manderson7]

We've got the cert problem solved we think, but are now getting timeouts using curl. I'm not seeing any activity in Splunk for the httplistener, but when they try from logstash, I get +

08-30-2018 09:43:09.193 -0400 WARN  HttpListener - Socket error from nattip while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request

and they see unable to fetch url.

I'm seeing traffic go through the FW, and tcpdump sees the traffic as well. What's stopping HEC from ingesting the data?

[harsmarvania57]

Are you sure that they are sending traffic over https and not http because while googling this error it says that this error generates when you send traffic over http

[manderson7]

So they were using http earlier, then switched to https and got the following:
[ERROR][logstash.outputs.http ] [HTTP Output Failure] Could not fetch URL {:url=>"https://nattdip:8088/services/collector", :method=>:post, :body=>"{\"@timestamp\":\"2018-08-30T15:00:05.110Z\",\"offset\":2617,\"@version\":\"1\",\"beat\":{\"name\":\"elk003.inf030.chi1.domain1.com\",\"hostname\":\"elk003.inf030.chi1.domain1.com\",\"version\":\"5.6.3\"},\"input_type\":\"log\",\"host\":\"elk003.inf030.chi1.domain1.com\",\"source\":\"/home/mbrien/splunk_test.log\",\"message\":\"tdedsadssdfst\",\"type\":\"splunk_test\",\"tags\":[\"test logs\",\"splunk\",\"beats_input_codec_plain_applied\"]}", :headers=>{"Authorization"=>"Splunk d82a61dd-xxxx-xxxx-xxxx-682bdcacb76e", "Content-Type"=>"application/json"}, :message=>"Connection reset", :class=>"Manticore::SocketException", :backtrace=>nil, :will_retry=>true}

[harsmarvania57]

Try with URL https://nattdip:8088/services/collector/raw, additionally try header like this headers => ['Authorization' , 'Splunk d82a61dd-xxxx-xxxx-xxxx-682bdcacb76e']

Unfortunately I don't have Logstash setup so I can't test in my lab environment but based on example given here https://discuss.elastic.co/t/logstash-to-splunk-http-event-collector/130765/3 it looks like that user is renaming message to event, which is require if you are sending data to https://URL:8088/services/collector , see example curl -k "https://mysplunkserver.example.com:8088/services/collector" \
-H "Authorization: Splunk CF179AE4-xxxx-xxxx-xxxx-328xxxxx67" \
-d '{"event": "Hello, world!", "sourcetype": "manual"}'

[manderson7]

Our firewall wasn't allowing the entire transmission through, as it's application based, and the wrong application was specified. We're able to transmit now, and ingest into Splunk. Thanks for your assistance in troubleshooting this.

[harsmarvania57]

It's good to hear that ELK is ingesting data in Splunk now. You can convert your last comment to answer and you can accept it.

[manderson7]

Considering a curl from one of my servers succeeded, but his failed, I'm asking our FW guys for some eyes before we attempt further on the logstash connection. Thanks very much for your help, and I'll come back to this when we get further in our connection.

[manderson7]

Thank you! Once I get them the appropriate cert we'll test this and let you know.