Getting Data In

How do I add a dynamic _meta field pushed from deployment server to monitored files on Heavy Fowarders?

Motivator

I am working with a heavy forwarder tier that is running syslog where network devices are sending data. For ease of tracking where each file is being monitored from, I would like to add some metadata to the monitored files, that includes the heavy forwarder they are being collected from (this tier is load balanced so the data could land on any number of hosts).

I have tried adding _meta = heavy_forwarder::$HOSTNAME and that does not appear to be doing the trick - It actually just puts $HOSTNAME as the value.

Example monitoring stanza:

[monitor:///var/log/remote/my_network_device]
index = network
sourcetype = mysourcetype
ignoreOlderThan = 1d
disabled = false
host_segment = 4
blacklist = \.(gz|tgz|xz|\d{1})$
_meta = heavy_forwarder::$HOSTNAME

I have also tried to add _meta = heavy_forwarder::$HOSTNAME to the [default] stanza and it has the same results. I know I can hardcode the hostname in system/local/inputs.com - I just am trying to think of a much less manual solution to handle dynamic scaling of the HF tier.

Any help would be greatly appreciated.

1 Solution

Influencer

Unfortunately, as of the writing of this post, Splunk does not have the ability to do templated configuration files... I actually have an open enhancement request for this sort of idea: SPL-136161 (Log a ticket to vote for it too). My particular use case was inserting tenant identifier to standardized index name patterns without crazy rewrite rules, but being able to have templated values in conf files would solve both.

As you already mention the possible solution of adding:

_meta = heavy_forwarder::actual.host.name

before all stanzas in etc/system/local/inputs.conf would make it apply to all input stanzas that do not specify their own _meta key (similar to how Splunk on FTR generates etc/system/local/inputs.conf and inserts host = actual.host.name and thus you could separate host specific configuration from the actual configuration, but agreed this is not as dynamic as I would like either.

View solution in original post

Influencer

Unfortunately, as of the writing of this post, Splunk does not have the ability to do templated configuration files... I actually have an open enhancement request for this sort of idea: SPL-136161 (Log a ticket to vote for it too). My particular use case was inserting tenant identifier to standardized index name patterns without crazy rewrite rules, but being able to have templated values in conf files would solve both.

As you already mention the possible solution of adding:

_meta = heavy_forwarder::actual.host.name

before all stanzas in etc/system/local/inputs.conf would make it apply to all input stanzas that do not specify their own _meta key (similar to how Splunk on FTR generates etc/system/local/inputs.conf and inserts host = actual.host.name and thus you could separate host specific configuration from the actual configuration, but agreed this is not as dynamic as I would like either.

View solution in original post

Motivator

What I have configured, which I do not like as it is not dynamic, is added:

_meta = heavy_forwarder::rsyslog.fqdn

To the /etc/system/local/inputs.conf file on each of the rsyslog hosts under the default stanza. I had to do this due to wanting to use the host regex on the monitor path for hostname. Also had to add the fields.conf on the IDXC/SHC members to be able to search for them properly. Don’t get me wrong, this works fine... It is just not as dynamic and requires me to start using puppet (which is ok, it is just another place for configurations that IMO should/could be managed by the deployment server dynamically via inputs.conf in specific inputs apps) to dynamically generate the rsyslog fqdn for the _meta tag I want to create in the system/local/inputs.conf file (per host) as new hosts are spun up.

This just seems like an oversite (not being able to use system variables in config files) - especially for systems running as Heavy Forwarders on Syslog hosts.

0 Karma

Path Finder

This is exactly what we looking for. A dynamic way to apply the hostname of our syslog servers to a custom indexed field. We're reduced to applying the following

_meta = field_name::host.name
0 Karma

Motivator

Just for clairification - I get $HOSTNAME in the field when I want it to be dynamic, so I don't have to set this up on system/local/inputs.conf on each host and hardcode the hostname.

0 Karma