We are developing Splunk dashboards. We have a Splunk enterprise server that is receiving HTTP event collector data form our clients. We have setup the server that receives the HEC information as a Heavy Forwarder that indexes locally and then forwards data to several other Splunk servers that we are using for development.
The servers we are forwarding to are losing events.
source=netmotion | stats count
for the same time span is returning significantly different count values. 13,089 from the splunk server receiving the HEC events, and 5,713 from the Splunk server we are forwarding to.
Why is this happening?
I understand, but if you can have a forwarder on the target server you could take logs from files, so you optimize log transmission and you're safer about log losing.
Otherwise, you should check if there could be a network problem during transmission.
I usually prefer to use Forwarders than HPPT Collector for the abome reasons.