Dashboards & Visualizations

It is possible to "tag" all data coming into a particular HEC token?

twinspop
Influencer

I have about 50 different tokens. I want data from one particular token to get some metadata added to it. Unfortunately, it doesn't appear that the _meta directive works for http in inputs.conf. Is it possible to replicate this functionality some how?

0 Karma

hrawat
Splunk Employee
Splunk Employee

Now you can tag HEC events for any HEC end point ( including s2s) without paying for third party software.

https://community.splunk.com/t5/Getting-Data-In/Splunk-HTTP-Event-Collector-support-for-custom-metad...

isoutamo
SplunkTrust
SplunkTrust
Update for old post as splunk has fixed this.
Currently (at lest 9.1.3+) you can use _meta also in HEC's inputs.conf.
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Have you tested if it works for both /raw and /event endpoints? Just asking because I haven't used it on HEC so I don't know 🙂

0 Karma

isoutamo
SplunkTrust
SplunkTrust
I test it at least for /raw endpoint.
0 Karma

ameizeraitis
New Member

You can use method i have implemented with DS distributed bash script automation, which does following with every single HEC input on each server in hfw pool:
First, append existing http stanzas in inputs.conf with "fake" output group, like

[http://hec_input_1]

outputgroup = out01

Define those fake outputs in outputs conf like this:

[tcpgroup:out01]

server=127.0.0.1:9001

Now we need to set some listener on internal loop input dedicated port that "tags" the data:

[splunktcp://9001]

_meta = HecName::192.168.0.1:hec_input_1

Repeat all this for for all your hec inputs, make each of it have it's own outputgroup and tcpsplunk port listener, restart splunk and enjoy:

|tstats count where index=hec_index by HecName

0 Karma

MuS
Legend

The inputs name will translate into a source::http:InputNameHere which in turn should be useable in props.conf
But I must admit, I have not yet tried it 😉

cheers, MuS

twinspop
Influencer

Well shoot. If the sending application sets source, that overrides the default above, which means the transform doesn't fire. So still back to the old problem: How to guarantee a transform gets applied to every single event that came through a particular token's input def?

0 Karma

MuS
Legend

In this case, did someone say cough cribl cough 😉

twinspop
Influencer

We're testing it, but not ready to roll into production. Yet. 🙂 Very promising!

0 Karma

twinspop
Influencer

Perfect! I had no idea that was a thing. I feel like I gained a new superpower.

0 Karma

MuS
Legend

Glad I could help - Enjoy the new superpower 🙂

cheers, MuS

0 Karma

MuS
Legend

Hi twinspop,

you can always use the good old props.conf / transforms.conf approach and add a meta field this way. Here is an example transforms.conf I use to add the hostname of the parsing HWF to events:

[add-relay-info-to-meta]
FORMAT = splunk_hwf::HostNameHere
REGEX = .
WRITE_META = true

Yes, it is a static value but I assume you will not change your HEC input too often 😉

Hope this helps ...

cheers, MuS

0 Karma

twinspop
Influencer

Yeah, a transform is where i was headed, but I don't see any foolproof way to identify only those those logs, and ALL those logs, that originate on 1 particular token. The token value and the input name are not things I can key off of in props as far as i know.

0 Karma
Get Updates on the Splunk Community!

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...