How to Forward events from logstash to Splunk, we are receiving an error

[msilva]

Hi,

For testing purposes, we are trying to use the Logstash client command line to send data to a Splunk server instance. The client configuration is the following (based on the steps found in https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Data/UsetheHTTPEventCollector) and it uses an HTTP Event Collector token:

input

{

      stdin

      {

            codec => json

      }

}

output

{

      http

      {

            format => "json"

            http_method => "post"

            url => "https://prd-p-kz4cj.splunkcloud.com:8088/services/collector/raw"

            headers => ["Authorization", "Splunk 26c964a2-c1e8-46e8-96ca-679d3b7542bd"]

      }

      

      stdout

      {

      }

}

However, it fails with the following error:

[ERROR] 2022-07-01 14:28:12.794 [[main]>worker5] http - Could not fetch URL {:url=>"https://prd-p-kz4cj.splunkcloud.com:8088/services/collector/raw", :method=>:post, :message=>"Connect to prd-p-kz4cj.splunkcloud.com:8088 [prd-p-kz4cj.splunkcloud.com/54.167.42.214] failed: connect timed out", :class=>Manticore::ConnectTimeout, :will_retry=>true}

And updating the url above to https://prd-p-kz4cj.splunkcloud.com/en-US:8088/services/collector/raw produces a different error:

[ERROR] 2022-07-01 14:34:26.702 [[main]>worker4] http - Encountered non-2xx HTTP code 404 {:response_code=>404, :url=>"https://prd-p-kz4cj.splunkcloud.com/en-US:8088/services/collector/raw", :event=>#<LogStash::Event:0x199312cb>}

Also, we have noticed that a ping to the Splunk server instance does not return a reply:

ping prd-p-kz4cj.splunkcloud.com

PING prd-p-kz4cj.splunkcloud.com (54.167.42.214) 56(84) bytes of data.

Could it be that a server configuration setting is missing?

Kind regards, Moacir Silva

[richgalloway]

Check your firewalls.  Connection timeouts often are causing by firewalls silently blocking connections.

[msilva]

Thank you for your reply,

It does not seem to be a firewall issue, since there should be no special rules defined in the related environment...

Also, we can successfully forward events to Elastic Security Solution SIEM using exactly the same approach.

Kind regards,

Moacir

[richgalloway]

Perhaps there *should* be no special rules, but have you checked if there *are* special rules?  Problems often are caused by the unexpected.

The ability to send to Elastic proves little.  Splunk is a different service listening on a different port, likely on a different server (should be a different server).  A difference in either server or port could change how firewalls handle the connection.

[msilva]

Hi,

Thank you, I will double-check the firewall rules and will let you know about it.

Kind regards,

Moacir

[msilva]

Hi,

I have installed locally a 60-day trial Splunk Enterprise Edition on http://127.0.0.1:8000/en-US/manager/launcher/http-eventcollector, created and enabled an HEC with a token value of f1039cf1-51c6-4efe-be4d-74b4c9245c0c but still a connection from the command line fails with: 

curl -u "x:f1039cf1-51c6-4efe-be4d-74b4c9245c0c" http://127.0.0.1:8088/services/collector/event -d '{"sourcetype": "my_sample_data", "event": "http auth ftw!"}'
curl: (56) Recv failure: Connection reset by peer

Maybe additional steps are needed?

Kind regards,

Moacir

[msilva]

I have managed to proceed further by disabling SSL in HEC global settings:

curl -u "x:f1039cf1-51c6-4efe-be4d-74b4c9245c0c" http://127.0.0.1:8088/services/collector/event -d '{"sourcetype": "my_sample_data", "event": "http auth ftw!"}'

{"text":"Success","code":0}

Howewer, it still fails when I try to send data through a Logstash client:

[ERROR] 2022-07-18 11:21:39.477 [[main]>worker6] http - Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://127.0.0.1:8088/services/collector/event", :event=>#<LogStash::Event:0x1d06c4f8>}

The related configuration file is the following:

input
{
stdin
{
codec => json
}
}

output
{
http
{
format => "json"
http_method => "post"
url => "http://127.0.0.1:8088/services/collector/event"
headers => ["Authorization", "Splunk f1039cf1-51c6-4efe-be4d-74b4c9245c0c"]
}

stdout
{
}
}

Kind regards,

Moacir

[Roy_9]

Even i faced the same issue and can only able to figure out by disabling SSL for HEC.

[msilva]

Ok, thank you.

For now, we will proceed without SSL.

Kind regards,

Moacir