How to Identify which HF is sending logs/metrics

[_pravin]

Hi,

I have incoming data from 2 Heavy Forwarders.

Both of forward HEC data and the internal logs, how do I identify which HF is sending a particular HEC data?

Regards,

Pravin

[gcusello]

Hi @_pravin ,

I di this and it runs:

# cat props.conf 
[default]
TRANSFORMS-default = set_splunk_hf

# cat transforms.conf 
[set_splunk_hf]
INGEST_EVAL = splunk_hf := splunk_server

# cat fields.conf
[splunk_hf]
INDEXED = true

Ciao.

Giuseppe

[PickleRick]

Remember that := operator is worse performance-wise than simple =

Also since you're using TRANSFORM, not RULESET, it will only be fired once on non-parsed data so unless you already have an _indexed field_ called splunk_hf with your unparsed data (a very very unlikely scenario), you can just use normal assignment operator.

[PickleRick]

Unless you explicitly do something to the data (for example, add an indexed field containing the name of the forwarder) Splunk doesn't keep this kind of metadata.

[_pravin]

I use Splunk Enterprise 9.0.4 and I tried adding _meta field which didn't work. I also tried adding INGEST_EVAL to transforms and tried sending the data, still no luck identifying the source.

[marnall]

Can you post how your _meta field was configured? It should be in inputs.conf and have the format:

_meta = fieldname::fieldvalue

So if you have two heavy forwarders, one can have an input with:

_meta = meta_hfnum::1

and the other:

_meta = meta_hfnum::2

[PickleRick]

Unfortunately, there is just one "instance" of _meta entry in the whole config. So you can't "merge" separate _meta settings - one will overwrite another. That's why TRANSFORMS is a better approach.

I'm also not sure what _meta will do on the splunktcp input especially when handling an input stream already containing metadata fields.

[marnall]

You can have separate _meta entries for different input stanzas. If you have two heavy forwarders handing different inputs then this should be doable. I've not tried it in a generalized input stanza but if the number of input stanzas are low then it is feasible to add _meta entries for each.

[PickleRick]

Yes. For different input stanzas - sure. But you can't - for example - have multiple apps defining multiple meta entries (like one for the environment the forwarder is in and another for the OS or team responsible or whatever) for the same input.

[dural_yyz]

I had done something like this in a previous life.  Each HF should get an app which has a props definition under the default stanza.  For a small number of HF's you can do this manually, for a large group to manage from like a DS reference the Splunk environment variables.

props.conf

[default]
splunk_forwarder = <HOSTNAME>

It has been a while so play around with this.  I seem to remember it was a props.conf mapped to transforms.conf which inserted the hostname so find what works the best for you. 

[dural_yyz]

Ok, it's been a long time but I finally had a use case to reimplement this from scratch in a new environment.

props.conf

[default]
TRANSFORMS-splunk_forwarder = splunk_forwarder_field

transforms.conf

[splunk_forwarder_field]
REGEX = .
WRITE_META = true
FORMAT = splunk_forwarder::text_host_name_of_choice

Note: The literal text after the :: becomes the value.

Additional value is I can reverence this field in a tstats by command to group things together.  It's helpful in many ways but I like be able to track count of ingested event by HF source for index source and sourctypes.

[PickleRick]

And how is this supposed to work? There is no property called splunk_forwarder in any props stanza. Also, Splunk does variable expansion on a very limited set of settings.

[hieuba6868]

you can check field call "splunk_server"

[_pravin]

Hi @hieuba6868 ,

I am sending open telemetry data to heavy forwarder and the HF forwards the data to indexers. When I look at the  field 'splunk_server' I can see only the name of indexers. If I look at the data I can see the name of the otel source. In my current scenario I want to know which is the HF sending the data.

Regards,

Pravin

[alemarzu]

Hi there, 

Not sure If you already did but the Monitoring Console could give you some insight. Mainly volume per token and activity by your HEC instances aka HFs.

Take a look under Indexing > Inputs > HTTP Event Collector: Instance