How to Identify which HF is sending logs/metrics

[_pravin]

Hi,

I have incoming data from 2 Heavy Forwarders.

Both of forward HEC data and the internal logs, how do I identify which HF is sending a particular HEC data?

Regards,

Pravin

[PickleRick]

Unless you explicitly do something to the data (for example, add an indexed field containing the name of the forwarder) Splunk doesn't keep this kind of metadata.

[_pravin]

I use Splunk Enterprise 9.0.4 and I tried adding _meta field which didn't work. I also tried adding INGEST_EVAL to transforms and tried sending the data, still no luck identifying the source.

[marnall]

Can you post how your _meta field was configured? It should be in inputs.conf and have the format:

_meta = fieldname::fieldvalue

So if you have two heavy forwarders, one can have an input with:

_meta = meta_hfnum::1

and the other:

_meta = meta_hfnum::2

[PickleRick]

Unfortunately, there is just one "instance" of _meta entry in the whole config. So you can't "merge" separate _meta settings - one will overwrite another. That's why TRANSFORMS is a better approach.

I'm also not sure what _meta will do on the splunktcp input especially when handling an input stream already containing metadata fields.

[marnall]

You can have separate _meta entries for different input stanzas. If you have two heavy forwarders handing different inputs then this should be doable. I've not tried it in a generalized input stanza but if the number of input stanzas are low then it is feasible to add _meta entries for each.

[PickleRick]

Yes. For different input stanzas - sure. But you can't - for example - have multiple apps defining multiple meta entries (like one for the environment the forwarder is in and another for the OS or team responsible or whatever) for the same input.

[dural_yyz]

I had done something like this in a previous life.  Each HF should get an app which has a props definition under the default stanza.  For a small number of HF's you can do this manually, for a large group to manage from like a DS reference the Splunk environment variables.

props.conf

[default]
splunk_forwarder = <HOSTNAME>

It has been a while so play around with this.  I seem to remember it was a props.conf mapped to transforms.conf which inserted the hostname so find what works the best for you. 

[PickleRick]

And how is this supposed to work? There is no property called splunk_forwarder in any props stanza. Also, Splunk does variable expansion on a very limited set of settings.

[hieuba6868]

you can check field call "splunk_server"

[_pravin]

Hi @hieuba6868 ,

I am sending open telemetry data to heavy forwarder and the HF forwards the data to indexers. When I look at the  field 'splunk_server' I can see only the name of indexers. If I look at the data I can see the name of the otel source. In my current scenario I want to know which is the HF sending the data.

Regards,

Pravin

[alemarzu]

Hi there, 

Not sure If you already did but the Monitoring Console could give you some insight. Mainly volume per token and activity by your HEC instances aka HFs.

Take a look under Indexing > Inputs > HTTP Event Collector: Instance