Getting Data In

Events missing via http event collector

phagunbaya
Explorer

I'm seeing a behaviour where some of my events are missing after been sent to http event collector. I'm sending single event per request. Sometimes it shows all the events and sometimes it does not. Normally it happens if the frequency is high (4-5 events per second).

splunker_2016_N
Engager

did this get fixed , even im missing a few events while running from java aws lambda

gblock_splunk
Splunk Employee
Splunk Employee

4 - 5 events per second is not high, we've designed HEC to support 100K a second on a single instance 🙂

  1. How big are the events?
  2. Are you getting a 200 OK response consistently?
  3. What is your configuration? Are you sending directly to HEC running on an indexer, or are you hosting HEC on a heavyweight forwarder and forwarding to the indexer?
0 Karma

netrc
New Member

I've seen HEC drop data with small events (100 bytes), sent 4-5/sec, for just a couple seconds (all just for testing). Each POST returns status 200 (OK). Tried this with both bash script using curl and also nodeJS; direct from the script to the HEC on the indexer. (Yes, the indexer is a little busy with other work). Over a couple hundred events, I've seen only 50% get stored.

But I'd think that getting a 200 (OK) would mean that the data is stored for sure in Splunk.

0 Karma

jplumsdaine22
Influencer

Are you getting a response from the indexer every time that the event was collected?

0 Karma

phagunbaya
Explorer

Yes. response was 200.

0 Karma

jkat54
SplunkTrust
SplunkTrust

This is a good point from jplumsdaine22... Do you confirm a 200 response and if not retry / fall back into an exception that can be handled?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...