topic Re: How to drop data from a HEC Collector? in Getting Data In

How to drop data from a HEC Collector?

skirven — Wed, 04 May 2022 16:31:35 GMT

Hi,
I'm trying to isolate why I'm not able to drop data from a HEC Collector endpoint. I have some docker logs I don't need to ingest. The Splunk HF is still on 7.3.8 for backwards compatibility, so I don't know if that's in play here. I checked with btool, and the files did load correctly.

inputs.conf:
- Sidenote here: When I set "source" value, it remained as "httpevent". But when I changed Sourcetype, the event changed correctly, which is odd.

[http://tpas_token] disabled = 0 index = elm-tpas-spc token = DD0D58D8-9F38-4A96-956C-XXXXXXXXXXXXXX source = tpas-event sourcetype = tpas-event

props.conf
- Sidenote: I tried also [ tpas-event ], and that also did not work

[ source::tpas-event ] TRANSFORMS-drop-handlers = drop-handlers

transforms.conf

[ drop-handlers ] REGEX = handlers.py|connection.py DEST_KEY = queue FORMAT = nullQueue

Re: How to drop data from a HEC Collector?

matthieup — Thu, 07 Sep 2023 05:11:50 GMT

Hello , did you find a solution for this problem ?
I'm facing the same issue and the data coming from HEC is never dropped.

Re: How to drop data from a HEC Collector?

PickleRick — Thu, 07 Sep 2023 10:41:53 GMT

The first question is - are your props/transforms on the same host as you're receiving data with your HEC input(s)? Or are you trying to receive HEC on HF and filter with props/transforms on indexers?

Re: How to drop data from a HEC Collector?

skirven — Thu, 07 Sep 2023 10:43:31 GMT

In my case, yes. Check your props.conf, and try with [httpevent] and see if that helps. I had to do a mixture of both to get it to work.

Re: How to drop data from a HEC Collector?

matthieup — Thu, 07 Sep 2023 10:55:39 GMT

I'm receiving the HEC directly on the search head and have the props/transforms setup on both the SH and the indexers.
The sourcetype is "jenkins_log" and the log I want to avoid has "DBCompilation" in the source field

This is what I'm trying to achieve.
in props.conf:

[jenkins_log] TRANSFORMS-override = ignore_jenkins_logs

in transforms.conf

[ignore_jenkins_logs] SOURCE_KEY = fields:source REGEX = DBCompilation DEST_KEY = queue FORMAT = nullQueue

Re: How to drop data from a HEC Collector?

PickleRick — Thu, 07 Sep 2023 11:36:00 GMT

Ugh. Using SH as an input collector is... kinda unusual. And not a very beautiful architecture.

Anyway, remember that your events are parsed and processed _only_ on the first "heavy" (based on the Splunk Enterprise install package; not UF) component in event's path (except ingest actions; those can happen on indexers even on parsed data). So if you're ingesting HEC on SH, you need those props/transforms on SH.

And in order to filter on the source field you need

MetaData:Source     : The source associated with the event.
                      The value must be prefixed by "source::"

Re: How to drop data from a HEC Collector?

matthieup — Thu, 07 Sep 2023 11:56:20 GMT

Noting your input on the SH being not the best option for the input collector of HEC.

Anyway, your tip was the correct one and allowed to filter the data. You made my day, thanks !