How to drop data from a HEC Collector?

[skirven]

Hi, 
   I'm trying to isolate why I'm not able to drop data from a HEC Collector endpoint. I have some docker logs I don't need to ingest. The Splunk HF is still on 7.3.8 for backwards compatibility, so I don't know if that's in play here. I checked with btool, and the files did load correctly.

inputs.conf:
 - Sidenote here: When I set "source" value, it remained as "httpevent". But when I changed Sourcetype, the event changed correctly, which is odd.

[http://tpas_token]
disabled = 0
index = elm-tpas-spc
token = DD0D58D8-9F38-4A96-956C-XXXXXXXXXXXXXX
source = tpas-event
sourcetype = tpas-event

props.conf
 - Sidenote: I tried also [ tpas-event ], and that also did not work

[ source::tpas-event ]
TRANSFORMS-drop-handlers = drop-handlers

 transforms.conf

[ drop-handlers ]
REGEX = handlers.py|connection.py
DEST_KEY = queue
FORMAT = nullQueue

[matthieup]

Hello , did you find a solution for this problem ?
I'm facing the same issue and the data coming from HEC is never dropped.

[PickleRick]

The first question is - are your props/transforms on the same host as you're receiving data with your HEC input(s)? Or are you trying to receive HEC on HF and filter with props/transforms on indexers?

[matthieup]

I'm receiving the HEC directly on the search head and have the props/transforms setup on both the SH and the indexers.
The sourcetype is "jenkins_log" and the log I want to avoid has "DBCompilation" in the source field

This is what I'm trying to achieve.
in props.conf:

[jenkins_log]
TRANSFORMS-override = ignore_jenkins_logs

in transforms.conf

[ignore_jenkins_logs]
SOURCE_KEY = fields:source
REGEX = DBCompilation
DEST_KEY = queue
FORMAT = nullQueue

[PickleRick]

Ugh. Using SH as an input collector is... kinda unusual. And not a very beautiful architecture.

Anyway, remember that your events are parsed and processed _only_ on the first "heavy" (based on the Splunk Enterprise install package; not UF) component in event's path (except ingest actions; those can happen on indexers even on parsed data). So if you're ingesting HEC on SH, you need those props/transforms on SH.

And in order to filter on the source field you need

MetaData:Source     : The source associated with the event.
                      The value must be prefixed by "source::"

[matthieup]

Noting your input on the SH being not the best option for the input collector of HEC.

Anyway, your tip was the correct one and allowed to filter the data. You made my day, thanks !

[skirven]

In my case, yes. Check your props.conf, and try with [httpevent]  and see if that helps. I had to do a mixture of both to get it to work.