topic Re: Intermediate forwarder auditing in Deployment Architecture

Intermediate forwarder auditing

cmeo-bcit — Wed, 12 Nov 2025 03:56:22 GMT

Consider email headers which show all the steps involved in getting the email item from where it was sent to where it is going, give or take spoofing and other whimsy.

Is there a search in Splunk to display all the systems which have handled a particular event, from the original forwarder (easy), via any and all intermediate forwarders (less so...).

The value of this would be to identify a possibly misconfigured system somewhere in what could be a complex chain, or where you have different intermediate forwarders depending on where you are in the network.

My feeling is that this is not logged, but I could be wrong, and even so, there may be a way to assemble the path an event has taken somehow.

Thanks!

Re: Intermediate forwarder auditing

PickleRick — Wed, 12 Nov 2025 07:19:26 GMT

You are right, by default Splunk doesn't capture this information.

You can use a ruleset (not transform since it would only be fired once) to add an indexed field to your event on each step along the way. But it only works on a "heavy" component.

Re: Intermediate forwarder auditing

gcusello — Wed, 12 Nov 2025 07:58:36 GMT

Hi @cmeo-bcit ,

I usually use this configuration (on HF) to create a field at index time (as @PickleRick hinted):

fields.conf:

[splunk_hf] INDEXED = true

props.conf:

[default] TRANSFORMS-default = set_splunk_hf

transforms.conf:

[set_splunk_hf] WRITE_META = true INGEST_EVAL = splunk_hf := splunk_server

This solution correctly runs if you have only one HF level; if your data pass through more HFs, you have to use a different transforms configuration:

[set_splunk_hf] WRITE_META = true INGEST_EVAL = splunk_hf=case(isnull(splunk_hf),splunk_server,1=1,splunk_hf.":".splunk_server)

I added a proposal on Splunk ideas to add this feature as a standard in Splunk systems and it's "under consideration", if you think that's a good idea, vote for it at https://ideas.splunk.com/ideas/EID-I-1731

Ciao.

Giuseppe

Re: Intermediate forwarder auditing

PickleRick — Wed, 12 Nov 2025 19:32:25 GMT

1. As I wrote before, transform will fire only once. Use a ruleset.

2. Use simple = instead of := in INGEST_EVAL. This will create multivalued fields. Yes. I tried doing it your way so that you have a single value with the whole chain but the := operator's performance is worse than straight assignment, it's easier to look for single values and the more intermediate steps you have, the more cardinality of your field raises.

3. I would be very cautious about using a transform (or ruleset) class called "default". It's a very popular word, it's easy to cause namespace clash.

Re: Intermediate forwarder auditing

cmeo-bcit — Sun, 16 Nov 2025 21:23:15 GMT

I've upvoted Giuseppe's Idea EID-I-1731 for this issue. In the meantime, the workarounds suggested will work for an HF only, which doesn't help if your intermediate forwarders are UF.

Accepted the answer anyway because this is as good as it gets for now.

Re: Intermediate forwarder auditing

gcusello — Mon, 17 Nov 2025 07:13:25 GMT

Hi @cmeo-bcit ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉