The problem is the value of the UF hostname the third party receives; it needs the hostname value to be the UF’s IP.
Splunk Cloud on the other hand needs the UF hostname to be in DNS format (as the host name is configured during first time boot in system/local/inputs.conf)
We thought about modifying inputs.conf such that the UFs had host=localhost, in combination with setting connection_host=(dns|ip) on Splunk Cloud and the HF, respectively. As seen in the documentation below.
We have verified that this would work for the third party, but would break the hostname in Splunk Cloud. Since the UFs have private IPs they cannot be dns-resolved by Splunk Cloud.
Our next idea is to rewrite the host values using transforms/props on the HF.
The problem is that the events being forwarded doesn’t contain the IP value of the UF host (Windows Event logs, can’t modify the raw event), so it’s not trivial to use a regex and write to MetaData::Host.
I know that the HF knows of the UF’s IP, as this can be seen in index=_internal source=*metrics.log sourceIP=*
Do somebody know of some (undocumented) way of tapping into that information and using it to overwrite the MetaData::Host value, such that the HF will output syslog as [timestamp][IP][event]?
* Receivers use this input stanza.
* This is the same as the [tcp://] stanza, except the remote server is assumed
to be a Splunk instance, most likely a forwarder.
* <remote server> is optional. If you specify it, the receiver only listen for
data from <remote server>.
* Use of <remote server is not recommended. Use the 'acceptFrom' setting,
which supersedes this setting.
connection_host = [ip|dns|none]
* For splunktcp, the 'host' or 'connection_host' will be used if the remote
Splunk instance does not set a host, or if the host is set to
* "ip" sets the host to the IP address of the system sending the data.
* "dns" sets the host to the reverse DNS entry for IP address of the system
sending the data.
* "none" leaves the host as specified in inputs.conf, typically the splunk
* Default: "ip".
I suggest a different approach: did you think to create a script to run on UF that writes UF's IP address in a file that is read by the UF and sent to the third party by HF?
In this way you haven't to transform hostname (that could be usefull) and at the same time you have the IP and you haven't DNS problems.