Splunk Cloud Platform

How to Forward events from logstash to Splunk, we are receiving an error

msilva
Path Finder

Hi,

For testing purposes, we are trying to use the Logstash client command line to send data to a Splunk server instance. The client configuration is the following (based on the steps found in https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Data/UsetheHTTPEventCollector) and it uses an HTTP Event Collector token:
 
input
{
      stdin
      {
            codec => json
      }
}
 
output
{
      http
      {
            format => "json"
            http_method => "post"
            url => "https://prd-p-kz4cj.splunkcloud.com:8088/services/collector/raw"
            headers => ["Authorization", "Splunk 26c964a2-c1e8-46e8-96ca-679d3b7542bd"]
      }
      
      stdout
      {
      }
}
 
 
However, it fails with the following error:
 
[ERROR] 2022-07-01 14:28:12.794 [[main]>worker5] http - Could not fetch URL {:url=>"https://prd-p-kz4cj.splunkcloud.com:8088/services/collector/raw", :method=>:post, :message=>"Connect to prd-p-kz4cj.splunkcloud.com:8088 [prd-p-kz4cj.splunkcloud.com/54.167.42.214] failed: connect timed out", :class=>Manticore::ConnectTimeout, :will_retry=>true}

 
And updating the url above to https://prd-p-kz4cj.splunkcloud.com/en-US:8088/services/collector/raw produces a different error:
 
[ERROR] 2022-07-01 14:34:26.702 [[main]>worker4] http - Encountered non-2xx HTTP code 404 {:response_code=>404, :url=>"https://prd-p-kz4cj.splunkcloud.com/en-US:8088/services/collector/raw", :event=>#<LogStash::Event:0x199312cb>}
 
 
Also, we have noticed that a ping to the Splunk server instance does not return a reply:
 
ping prd-p-kz4cj.splunkcloud.com
PING prd-p-kz4cj.splunkcloud.com (54.167.42.214) 56(84) bytes of data.
 
 
Could it be that a server configuration setting is missing?
Kind regards,
 
 
 
Moacir Silva
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Check your firewalls.  Connection timeouts often are causing by firewalls silently blocking connections.

---
If this reply helps you, Karma would be appreciated.
0 Karma

msilva
Path Finder

Thank you for your reply,

It does not seem to be a firewall issue, since there should be no special rules defined in the related environment...

Also, we can successfully forward events to Elastic Security Solution SIEM using exactly the same approach.

Kind regards,

 

Moacir

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Perhaps there *should* be no special rules, but have you checked if there *are* special rules?  Problems often are caused by the unexpected.

The ability to send to Elastic proves little.  Splunk is a different service listening on a different port, likely on a different server (should be a different server).  A difference in either server or port could change how firewalls handle the connection.

---
If this reply helps you, Karma would be appreciated.
0 Karma

msilva
Path Finder

Hi,

Thank you, I will double-check the firewall rules and will let you know about it.

Kind regards,

 

Moacir

0 Karma

msilva
Path Finder

Hi,

I have installed locally a 60-day trial Splunk Enterprise Edition on http://127.0.0.1:8000/en-US/manager/launcher/http-eventcollector, created and enabled an HEC with a token value of f1039cf1-51c6-4efe-be4d-74b4c9245c0c but still a connection from the command line fails with: 

 

curl -u "x:f1039cf1-51c6-4efe-be4d-74b4c9245c0c" http://127.0.0.1:8088/services/collector/event -d '{"sourcetype": "my_sample_data", "event": "http auth ftw!"}'
curl: (56) Recv failure: Connection reset by peer

 

Maybe additional steps are needed?

Kind regards,

 

Moacir

0 Karma

msilva
Path Finder

I have managed to proceed further by disabling SSL in HEC global settings:

curl -u "x:f1039cf1-51c6-4efe-be4d-74b4c9245c0c" http://127.0.0.1:8088/services/collector/event -d '{"sourcetype": "my_sample_data", "event": "http auth ftw!"}'
{"text":"Success","code":0}

Howewer, it still fails when I try to send data through a Logstash client:

[ERROR] 2022-07-18 11:21:39.477 [[main]>worker6] http - Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://127.0.0.1:8088/services/collector/event", :event=>#<LogStash::Event:0x1d06c4f8>}

The related configuration file is the following:

input
{
stdin
{
codec => json
}
}

output
{
http
{
format => "json"
http_method => "post"
url => "http://127.0.0.1:8088/services/collector/event"
headers => ["Authorization", "Splunk f1039cf1-51c6-4efe-be4d-74b4c9245c0c"]
}

stdout
{
}
}

Kind regards,

 

Moacir

Roy_9
Motivator

Even i faced the same issue and can only able to figure out by disabling SSL for HEC.

0 Karma

msilva
Path Finder

Ok, thank you.

For now, we will proceed without SSL.

Kind regards,

 

Moacir

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...