topic Re: It is possible to "tag" all data coming into a particular HEC token? in Dashboards & Visualizations

It is possible to "tag" all data coming into a particular HEC token?

twinspop — Mon, 22 Jul 2019 19:53:27 GMT

I have about 50 different tokens. I want data from one particular token to get some metadata added to it. Unfortunately, it doesn't appear that the _meta directive works for http in inputs.conf. Is it possible to replicate this functionality some how?

Re: It is possible to "tag" all data coming into a particular HEC token?

MuS — Mon, 22 Jul 2019 20:16:53 GMT

Hi twinspop,

you can always use the good old props.conf / transforms.conf approach and add a meta field this way. Here is an example transforms.conf I use to add the hostname of the parsing HWF to events:

[add-relay-info-to-meta]
FORMAT = splunk_hwf::HostNameHere
REGEX = .
WRITE_META = true

Yes, it is a static value but I assume you will not change your HEC input too often 😉

Hope this helps ...

cheers, MuS

Re: It is possible to "tag" all data coming into a particular HEC token?

twinspop — Mon, 22 Jul 2019 20:23:05 GMT

Yeah, a transform is where i was headed, but I don't see any foolproof way to identify only those those logs, and ALL those logs, that originate on 1 particular token. The token value and the input name are not things I can key off of in props as far as i know.

Re: It is possible to "tag" all data coming into a particular HEC token?

MuS — Mon, 22 Jul 2019 20:49:47 GMT

The inputs name will translate into a source::http:InputNameHere which in turn should be useable in props.conf
But I must admit, I have not yet tried it 😉

cheers, MuS

Re: It is possible to "tag" all data coming into a particular HEC token?

twinspop — Mon, 22 Jul 2019 21:04:18 GMT

Perfect! I had no idea that was a thing. I feel like I gained a new superpower.

Re: It is possible to "tag" all data coming into a particular HEC token?

MuS — Mon, 22 Jul 2019 21:06:57 GMT

Glad I could help - Enjoy the new superpower 🙂

cheers, MuS

Re: It is possible to "tag" all data coming into a particular HEC token?

twinspop — Mon, 22 Jul 2019 21:08:28 GMT

Well shoot. If the sending application sets source, that overrides the default above, which means the transform doesn't fire. So still back to the old problem: How to guarantee a transform gets applied to every single event that came through a particular token's input def?

Re: It is possible to "tag" all data coming into a particular HEC token?

MuS — Mon, 22 Jul 2019 21:17:09 GMT

In this case, did someone say cough cribl cough 😉

Re: It is possible to "tag" all data coming into a particular HEC token?

twinspop — Mon, 22 Jul 2019 21:30:22 GMT

We're testing it, but not ready to roll into production. Yet. 🙂 Very promising!

Re: It is possible to "tag" all data coming into a particular HEC token?

ameizeraitis — Tue, 16 Aug 2022 13:55:30 GMT

You can use method i have implemented with DS distributed bash script automation, which does following with every single HEC input on each server in hfw pool:
First, append existing http stanzas in inputs.conf with "fake" output group, like

[http://hec_input_1]

outputgroup = out01

Define those fake outputs in outputs conf like this:

[tcpgroup:out01]

server=127.0.0.1:9001

Now we need to set some listener on internal loop input dedicated port that "tags" the data:

[splunktcp://9001]

_meta = HecName::192.168.0.1:hec_input_1

Repeat all this for for all your hec inputs, make each of it have it's own outputgroup and tcpsplunk port listener, restart splunk and enjoy:

|tstats count where index=hec_index by HecName

Re: It is possible to "tag" all data coming into a particular HEC token?

isoutamo — Fri, 17 May 2024 14:48:13 GMT

Update for old post as splunk has fixed this.
Currently (at lest 9.1.3+) you can use _meta also in HEC's inputs.conf.

Re: It is possible to "tag" all data coming into a particular HEC token?

PickleRick — Fri, 17 May 2024 21:07:46 GMT

Have you tested if it works for both /raw and /event endpoints? Just asking because I haven't used it on HEC so I don't know 🙂

Re: It is possible to "tag" all data coming into a particular HEC token?

isoutamo — Fri, 31 May 2024 10:47:23 GMT

I test it at least for /raw endpoint.

Re: It is possible to "tag" all data coming into a particular HEC token?

hrawat — Mon, 04 Nov 2024 21:36:26 GMT

Now you can tag HEC events for any HEC end point ( including s2s) without paying for third party software.

https://community.splunk.com/t5/Getting-Data-In/Splunk-HTTP-Event-Collector-support-for-custom-metadata-tags/m-p/703131/highlight/true#M116292