How do you track down HEC senders with bad indexes?

[twinspop]

One of my defined HEC tokens is receiving a lot more traffic than it's writing to indexes. I'm comparing the indexes it has access to and the introspection stats for the event collector. The EC stats are showing upwards of 3 MB/s. The indexes have no where near that volume. I suspect someone is sending using that token and a bad index. Indeed, I have these logs showing up at nearly 1 per/second:

03-08-2019 16:08:59.964 -0500 ERROR HttpInputDataHandler - Parsing error : Incorrect index

But that log is worthless. Index? Host or IP? Source? Sourcetype? Anything?! I also tried enabling debug for HttpInputDataHandler but it didn't advance my quest any.

How to track this down aside from tcpdump and hours of hunting through packet captures?

[dperre_splunk]

Just in case someone comes across this issue in the future. This is how I solved the problem.

Enabled debug for the HttpInputDataHandler log file.

Searched for the following
index=_internal HttpInputDataHandler source="/opt/splunk/var/log/splunk/splunkd.log" "reply: 7"

This showed me the source ip address that was making the call.

Reply 7 is incorrect index as referred in the following article
https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/TroubleshootHTTPEventCollector#Possible_erro...

[maciep]

i wonder if maybe you can use the tcpin logs on the hec server to correlate over time/counts of the incorrect index events?

but i agree, that log line is mostly useless and you would think hec at least knows the index and hopefully some other metadata too...just log it.

[twinspop]

Good idea! BUT, we have a load balancer in front of HEC. All the IPs are the same.

[maciep]

do you have the lb logs in splunk? in my env, i would look at the translated dest to find the hec and source for the originating ip. I've never tried tracking this problem, but those are the lb fields i'd use if I wanted to try....if you have them of course.

[rashi83]

@twinspop : Interested to know if you found the issue? I am also facing the same issue - there is no LB infront of HEC.