Getting Data In

Heavy Forwarder losing data when forwarding data from HTTP Event Collector

simpkins1958
Contributor

We are developing Splunk dashboards. We have a Splunk enterprise server that is receiving HTTP event collector data form our clients. We have setup the server that receives the HEC information as a Heavy Forwarder that indexes locally and then forwards data to several other Splunk servers that we are using for development.

The servers we are forwarding to are losing events.

source=netmotion | stats count

for the same time span is returning significantly different count values. 13,089 from the splunk server receiving the HEC events, and 5,713 from the Splunk server we are forwarding to.

Why is this happening?

0 Karma

simpkins1958
Contributor

We had outputs.conf configured wrong. We were auto load balancing to the three servers.

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

@simpkins1958, did that resolve the issue? If so I will close the question.

0 Karma

simpkins1958
Contributor

Yes. User error...

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi simpkins1958,
can you install a Forwarder on the target systems?
If you can, use a Forwarder to ingest and send logs to the Indexer, to be safer about log losing.
Bye.
Giuseppe

0 Karma

simpkins1958
Contributor

We are not getting data from log files. We are getting data from our clients through the Splunk HTTP Event Collector.

0 Karma

gcusello
SplunkTrust
SplunkTrust

I understand, but if you can have a forwarder on the target server you could take logs from files, so you optimize log transmission and you're safer about log losing.
Otherwise, you should check if there could be a network problem during transmission.
I usually prefer to use Forwarders than HPPT Collector for the abome reasons.
Bye.
Giuseppe

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...