I'm seeing a behaviour where some of my events are missing after been sent to http event collector. I'm sending single event per request. Sometimes it shows all the events and sometimes it does not. Normally it happens if the frequency is high (4-5 events per second).
4 - 5 events per second is not high, we've designed HEC to support 100K a second on a single instance 🙂
I've seen HEC drop data with small events (100 bytes), sent 4-5/sec, for just a couple seconds (all just for testing). Each POST returns status 200 (OK). Tried this with both bash script using curl and also nodeJS; direct from the script to the HEC on the indexer. (Yes, the indexer is a little busy with other work). Over a couple hundred events, I've seen only 50% get stored.
But I'd think that getting a 200 (OK) would mean that the data is stored for sure in Splunk.