Getting Data In

How to drop data from a HEC Collector?

skirven
Communicator

Hi, 
   I'm trying to isolate why I'm not able to drop data from a HEC Collector endpoint. I have some docker logs I don't need to ingest. The Splunk HF is still on 7.3.8 for backwards compatibility, so I don't know if that's in play here. I checked with btool, and the files did load correctly.

inputs.conf:
 - Sidenote here: When I set "source" value, it remained as "httpevent". But when I changed Sourcetype, the event changed correctly, which is odd.

 

 

[http://tpas_token]
disabled = 0
index = elm-tpas-spc
token = DD0D58D8-9F38-4A96-956C-XXXXXXXXXXXXXX
source = tpas-event
sourcetype = tpas-event

 

 

props.conf
 - Sidenote: I tried also [ tpas-event ], and that also did not work

 

 

[ source::tpas-event ]
TRANSFORMS-drop-handlers = drop-handlers

 

 

 transforms.conf

 

 

[ drop-handlers ]
REGEX = handlers.py|connection.py
DEST_KEY = queue
FORMAT = nullQueue

 

 

 

Labels (2)
Tags (1)
0 Karma

matthieup
Explorer

Hello , did you find a solution for this problem ?
I'm facing the same issue and the data coming from HEC is never dropped.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The first question is - are your props/transforms on the same host as you're receiving data with your HEC input(s)? Or are you trying to receive HEC on HF and filter with props/transforms on indexers?

matthieup
Explorer

I'm receiving the HEC directly on the search head and have the props/transforms setup on both the SH and the indexers.
The sourcetype is "jenkins_log" and the log I want to avoid has "DBCompilation" in the source field

This is what I'm trying to achieve.
in props.conf:

 

 

[jenkins_log]
TRANSFORMS-override = ignore_jenkins_logs

 



in transforms.conf

 

[ignore_jenkins_logs]
SOURCE_KEY = fields:source
REGEX = DBCompilation
DEST_KEY = queue
FORMAT = nullQueue

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Ugh. Using SH as an input collector is... kinda unusual. And not a very beautiful architecture.

Anyway, remember that your events are parsed and processed _only_ on the first "heavy" (based on the Splunk Enterprise install package; not UF) component in event's path (except ingest actions; those can happen on indexers even on parsed data). So if you're ingesting HEC on SH, you need those props/transforms on SH.

And in order to filter on the source field you need

MetaData:Source     : The source associated with the event.
                      The value must be prefixed by "source::"

matthieup
Explorer

Noting your input on the SH being not the best option for the input collector of HEC.

Anyway, your tip was the correct one and allowed to filter the data. You made my day, thanks !

0 Karma

skirven
Communicator

In my case, yes. Check your props.conf, and try with [httpevent]  and see if that helps. I had to do a mixture of both to get it to work.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...