topic Re: How to Identify which HF is sending logs/metrics in Getting Data In

How to Identify which HF is sending logs/metrics

_pravin — Tue, 12 Nov 2024 13:51:36 GMT

Hi,

I have incoming data from 2 Heavy Forwarders.

Both of forward HEC data and the internal logs, how do I identify which HF is sending a particular HEC data?

Regards,

Pravin

Re: How to Identify which HF is sending logs/metrics

hieuba6868 — Tue, 12 Nov 2024 19:44:43 GMT

you can check field call "splunk_server"

Re: How to Identify which HF is sending logs/metrics

PickleRick — Tue, 12 Nov 2024 20:05:34 GMT

Unless you explicitly do something to the data (for example, add an indexed field containing the name of the forwarder) Splunk doesn't keep this kind of metadata.

Re: How to Identify which HF is sending logs/metrics

_pravin — Wed, 13 Nov 2024 10:24:49 GMT

Hi @hieuba6868 ,

I am sending open telemetry data to heavy forwarder and the HF forwards the data to indexers. When I look at the field 'splunk_server' I can see only the name of indexers. If I look at the data I can see the name of the otel source. In my current scenario I want to know which is the HF sending the data.

Regards,

Pravin

Re: How to Identify which HF is sending logs/metrics

_pravin — Wed, 13 Nov 2024 10:26:45 GMT

I use Splunk Enterprise 9.0.4 and I tried adding _meta field which didn't work. I also tried adding INGEST_EVAL to transforms and tried sending the data, still no luck identifying the source.

Re: How to Identify which HF is sending logs/metrics

alemarzu — Fri, 22 Nov 2024 15:47:05 GMT

Hi there,

Not sure If you already did but the Monitoring Console could give you some insight. Mainly volume per token and activity by your HEC instances aka HFs.

Take a look under Indexing > Inputs > HTTP Event Collector: Instance

Re: How to Identify which HF is sending logs/metrics

dural_yyz — Fri, 22 Nov 2024 17:55:43 GMT

I had done something like this in a previous life. Each HF should get an app which has a props definition under the default stanza. For a small number of HF's you can do this manually, for a large group to manage from like a DS reference the Splunk environment variables.

props.conf

[default]
splunk_forwarder = <HOSTNAME>

It has been a while so play around with this. I seem to remember it was a props.conf mapped to transforms.conf which inserted the hostname so find what works the best for you.

Re: How to Identify which HF is sending logs/metrics

PickleRick — Fri, 22 Nov 2024 20:46:47 GMT

And how is this supposed to work? There is no property called splunk_forwarder in any props stanza. Also, Splunk does variable expansion on a very limited set of settings.

Re: How to Identify which HF is sending logs/metrics

marnall — Sat, 23 Nov 2024 15:20:44 GMT

Can you post how your _meta field was configured? It should be in inputs.conf and have the format:

_meta = fieldname::fieldvalue

So if you have two heavy forwarders, one can have an input with:

_meta = meta_hfnum::1

and the other:

_meta = meta_hfnum::2

Re: How to Identify which HF is sending logs/metrics

PickleRick — Sat, 23 Nov 2024 15:34:14 GMT

Unfortunately, there is just one "instance" of _meta entry in the whole config. So you can't "merge" separate _meta settings - one will overwrite another. That's why TRANSFORMS is a better approach.

I'm also not sure what _meta will do on the splunktcp input especially when handling an input stream already containing metadata fields.

Re: How to Identify which HF is sending logs/metrics

marnall — Sat, 23 Nov 2024 15:52:17 GMT

You can have separate _meta entries for different input stanzas. If you have two heavy forwarders handing different inputs then this should be doable. I've not tried it in a generalized input stanza but if the number of input stanzas are low then it is feasible to add _meta entries for each.

Re: How to Identify which HF is sending logs/metrics

PickleRick — Sat, 23 Nov 2024 18:30:17 GMT

Yes. For different input stanzas - sure. But you can't - for example - have multiple apps defining multiple meta entries (like one for the environment the forwarder is in and another for the OS or team responsible or whatever) for the same input.

Re: How to Identify which HF is sending logs/metrics

dural_yyz — Wed, 21 Jan 2026 23:10:50 GMT

Ok, it's been a long time but I finally had a use case to reimplement this from scratch in a new environment.

props.conf

[default] TRANSFORMS-splunk_forwarder = splunk_forwarder_field

transforms.conf

[splunk_forwarder_field] REGEX = . WRITE_META = true FORMAT = splunk_forwarder::text_host_name_of_choice

Note: The literal text after the :: becomes the value.

Additional value is I can reverence this field in a tstats by command to group things together. It's helpful in many ways but I like be able to track count of ingested event by HF source for index source and sourctypes.

Re: How to Identify which HF is sending logs/metrics

gcusello — Wed, 21 Jan 2026 16:57:18 GMT

Hi @_pravin ,

I di this and it runs:

# cat props.conf [default] TRANSFORMS-default = set_splunk_hf # cat transforms.conf [set_splunk_hf] INGEST_EVAL = splunk_hf := splunk_server # cat fields.conf [splunk_hf] INDEXED = true

Ciao.

Giuseppe

Re: How to Identify which HF is sending logs/metrics

PickleRick — Thu, 22 Jan 2026 17:44:28 GMT

Remember that := operator is worse performance-wise than simple =

Also since you're using TRANSFORM, not RULESET, it will only be fired once on non-parsed data so unless you already have an _indexed field_ called splunk_hf with your unparsed data (a very very unlikely scenario), you can just use normal assignment operator.